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Preface 


We live in a world that is, to a large extent, shaped by technology. Our computers and our 
smartphones: technology. Our automobiles and airplanes: technology. Our homes and offices, 
our food and clothing, our heating and cooling, our entertainment, our medical care: They are all 
created and driven by technology. 

Likewise, many of the critical challenges that we face as a society, and that our young people 
will eventually need to address, have large technological components such as the quest to li nk 
experts throughout the world, the search for sustainable energy, dealing with global pandemics, 
and the development of environmentally friendly agriculture to feed a growing world population. 

Despite its importance, however, technology has not been a focus of instruction and assessment 
in our educational system, particularly at the elementary and secondary levels. Some 
technologies are being integrated into humanities, social science, science, and mathematics 
classes as methods for supporting learning in these subjects. Technologies are also becoming 
subjects of study, with the goal of developing an understanding of the technologies themselves, 
their various roles, and the engineering design processes to address human needs and wants. 
However, there are no standardized, national assessments to provide evidence of what 
technologically literate K-12 students know about technology and engineering and the roles they 
play in our lives, or the extent to which students can use technologies and understand how 
engineers design and develop them. 

Technology and engineering are increasingly being incorporated into school coursework, ranging 
from instruction on the use of computers and information technology within school subjects to 
classes that examine the role of technology in society, or courses that teach engineering design. 
Infonnation communication technologies have become integral tools of the trade in academic, 
workplace, and practical contexts. Technology and engineering are essential components of 
contemporary science, technology, engineering, and mathematics (STEM) education. 

Because of this growing importance of technology and engineering in the educational landscape, 
the National Assessment Governing Board decided that an assessment of technological literacy 
would be an important addition to the National Assessment of Educational Progress (NAEP). For 
more than 35 years, NAEP has assessed achievement in a variety of key subjects by testing 
samples of students most often in the 4th, 8th, and 12th grades. The results, commonly referred 
to as The Nation’s Report Card, have become an important source of information on what U.S. 
students know and are able to do in a range of subject areas. 

The Governing Board sought a framework of technological literacy knowledge and skills that 
identifies the understandings and applications of technology principles that are important for all 
students. The framework focuses on “literacy” as the level of knowledge and competencies 
needed by all students and citizens. People who are literate about technology and engineering are 
not expected to “do” engineering or produce technology in the professional sense. Therefore, the 
framework is not intended to address technical knowledge of specific technologies or types of 
engineering expertise taught in specialized courses to prepare some students for postsecondary 
engineering studies. At grade 4, for example, all students can be expected to identify types of 
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technologies in their world, design and test a simple model, explain how technologies can result 
in positive and negative effects, and use common technologies to achieve goals in school and 
their everyday life. By grade 12, for example, all students can be expected to select and use a 
variety of tools and media to conduct research, evaluate how well a solution meets specified 
criteria, and develop a plan to address a complex global issue. 

The Governing Board chose the contractor WestEd to recommend a framework for this new 
assessment. WestEd in turn assembled a broad array of individuals and organizations to take part 
in the effort. 

The resulting framework is the culmination of a long, complex process that drew on the 
contributions of thousands of individuals and organizations with expertise in the use of 
technology for learning, technology education, and engineering. Eighteen outreach meetings held 
over a 15 -month period gathered feedback from professional organizations, practitioners, and the 
general public. Surveys documented comments on drafts of the framework. 

As the framework was being developed, it became clear that the tenns “technology,” 
“engineering,” “information communication technology,” “21st-century skills,” and “literacy” 
are defined and used in significantly different ways in formal and infonnal education, in 
standards, by professional organizations, and in legislation. Therefore, the framework 
development committees recommended a change of the framework title from “technological 
literacy” to “technology and engineering literacy” to encompass general literacy about the use, 
effects, and designing of technologies. The 2014 NAEP Technology and Engineering Literacy 
Framework is a statement about what should be expected of students in terms of their knowledge 
and skills with technology, written to be the basis for an assessment of technology and 
engineering literacy appropriate for all students. It opens the door to seeing what our K-12 
students know about technology and engineering, in the same way that NAEP assesses their 
knowledge and capabilities in reading, mathematics, science, and other subjects. 

This framework describes the assessment of technology and engineering literacy at grades 4, 8, 
and 12, although not all 3 grades will be included in the initial assessment. The 2014 NAEP 
Technology and Engineering Literacy Assessment is planned as a probe. In the NAEP context, a 
probe is a smaller-scale, focused assessment on a timely topic that explores a particular question 
or issue and may be limited to particular grades. 

Technology and Engineering Literacy 

Any assessment of students’ technology and engineering literacy must start with a clear idea of 
exactly what technology and engineering literacy means. That in turn requires clear definitions of 
technology and engineering. 

Research shows that most Americans associate technology with computers and related electronic 
devices. However, while the computer is certainly an important example of technology — and one 
that plays an especially important role in this framework — historically the term “technology” has 
had a much broader and deeper meaning, and it is this meaning that is represented in this 
framework: 
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• Technology is any modification of the natural world done to fulfill human needs or 
desires. 

This definition sees technology as encompassing the entire human-made world, from the 
simplest artifacts, such as paper and pencil, to the most complex — buildings and cities, the 
electric power grid, satellites, and the Internet. Furthermore, technology is not just the things that 
people create. It includes the entire infrastructure needed to design, manufacture, operate, and 
repair technological artifacts, from corporate headquarters and engineering schools to 
manufacturing plants, media outlets, and distribution networks. 

Engineers are the agents for designing the many technologies that modify the world. Engineers 
may not actually construct artifacts such as calculators, bridges, or airplanes. Engineers develop 
the plans and directions for how artifacts are to be constructed, as well as design processes such 
as assembly lines or procedures for clinical trials of phannaceuticals. The framework uses the 
following definition of engineering: 

• Engineering is a systematic and often iterative approach to designing objects, 
processes, and systems to meet human needs and wants. 

This framework defines technology and engineering literacy in a broad fashion: 

• Technology and engineering literacy is the capacity to use, understand, and 
evaluate technology as well as to understand technological principles and 
strategies needed to develop solutions and achieve goals. 

Thus, technology and engineering literacy has much in common with scientific, mathematical, 
and language literacy. As with these other forms of academic literacy, technology and 
engineering literacy involves the mastery of a set of tools needed to participate intelligently and 
thoughtfully in society. The tools are different, but the ultimate goal is the same. One particularly 
important set of technological tools consists of information and communication technologies, 
and this framework sees knowledge of these particular technologies as an integral part of 
technology and engineering literacy. 

Content 

This framework describes the essential knowledge and skills that will be assessed on the NAEP 
Technology and Engineering Literacy Assessment, beginning in 2014. It is not possible to assess 
every facet of technology and engineering literacy — the field is far too broad — so the assessment 
targets have been selected to capture those aspects of the nature, processes, uses, and effects of 
technology that are particularly important to participation in the economic, civic, and social 
spheres of modern society. 
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Areas of Technology and Engineering Literacy 

The assessment targets are organized into three 
major areas of technology and engineering 
literacy, each corresponding to a broad body of 
knowledge and skills with which students should 
be familiar: 

• Technology and Society 

• Design and Systems 

• Infonnation and Communication 
Technology 

Technology and Society deals with the effects 
that technology has on society and the natural 
world and with the sorts of ethical questions that 
arise from those effects. Knowledge and 
capabilities in this area are crucial for 
understanding the issues surrounding the development and use of various technologies and for 
participating in decisions regarding their use. 

Design and Systems covers the nature of technology, the engineering design process by which 
technologies are developed, and basic principles of dealing with everyday technologies, 
including maintenance and troubleshooting. An understanding of the design process is 
particularly valuable in assessing technologies, and it can also be applied in areas outside 
technology, since design is a broadly applicable skill. 

Information and Communication Technology includes computers and software learning tools, 
networking systems and protocols, hand-held digital devices, and other technologies for 
accessing, creating, and communicating information and for facilitating creative expression. 
Although it is just one among several types of technologies, it has achieved a special prominence 
in technology and engineering literacy because familiarity and facility with it is essential in 
virtually every profession in modern society. 

These three areas of technology and engineering literacy are closely interconnected. For 
example, to address an issue related to technology and society such as clean water, energy needs, 
digital-age communications, or global climate change, it is important to have an understanding of 
technological systems and the design process. Similarly, one must also be able to use various 
information and communication technologies to research problems, develop and communicate 
possible solutions, and achieve goals. 

Practices 

In all three areas of technology and engineering literacy, students are expected to be able to 
apply particular ways of thinking and reasoning when approaching a problem. These types of 
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thinking and reasoning are referred to as practices, and the framework specifies three kinds in 
particular that students are expected to demonstrate when responding to test questions: 

• Understanding Technological Principles 

• Developing Solutions and Achieving Goals 

• Communicating and Collaborating 

Understanding Technological Principles focuses on how well students are able to make use of 
their knowledge about technology. This knowledge ranges from discrete declarative facts and 
concepts to higher-level reasoning about how facts, concepts, and principles are organized into 
various structures and relationships. Students should be able to use their knowledge about 
technology to develop explanations and make predictions, comparisons, and evaluations. 

Developing Solutions and Achieving Goals refers to students’ systematic use of technological 
knowledge, tools, and skills to solve problems and achieve goals presented in realistic contexts. 
This practice includes both procedural and strategic capabilities — knowing how to apply simple 
steps and use technological tools to address real situations, as well as when and where to apply 
the tools and processes. 

Communicating and Collaborating concerns how well students are able to use contemporary 
technologies to communicate for a variety of purposes and in a variety of ways, working 
individually or in teams, with peers and experts. 

Contexts 

Technology and engineering literacy requires not just that students know about technology but 
also that they are able to recognize the technologies around them and apply what they know to 
problems and projects involving particular technologies. Consequently, NAEP Technology and 
Engineering Literacy Assessment items will measure students’ technology and engineering 
literacy in the contexts of relevant societal issues and of actual problems that people are 
commonly called on to solve and goals they seek to achieve. 

The contexts will vary according to the area of technology and engineering literacy and the 
practice being assessed, but they should include a broad sampling from the major technological 
fields in use today: medical technologies, information and communication technologies, energy 
and power technologies, transportation technologies, agriculture and biotechnology, and so forth. 

Types of Tasks and Items 

Technology and engineering literacy involves a range of knowledge and capabilities whose 
assessment requires having students perform a variety of actions using diverse tools to solve 
problems and meet goals within rich, complex scenarios that reflect realistic situations. To learn 
what students know and can do with regard to technology and engineering, the framework calls 
for the assessment to be totally computer-based. The use of computers makes it feasible to utilize 
scenario-based multimedia tasks developed for the assessment, along with more traditional item 
types. Although many items will be selected-response items, such as multiple choice, in which 
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only the final answer is observed, there will also be a number of more complex items that allow 
the monitoring of the students’ actions as they manipulate components of the systems and 
models that are presented to them. 

Reporting Results 

Results of the NAEP Technology and Engineering Literacy Assessment will be reported in terms 
of percentages of students who attain each of the three achievement levels — Basic, Proficient, 
and Advanced — as is typical for NAEP results in other subjects. In addition, results of the 
assessment will be reported in terms of NAEP scale scores, on a 0-500 scale, for example. Scale 
scores will be reported for each of the three major assessment areas of technology and 
engineering literacy: Technology and Society, Design and Systems, and Information and 
Communication Technology. An overall composite score will also be reported. 

In addition to scores, data will be collected on student gender, race/ethnicity, eligibility for free 
and reduced-price lunch, disabilities, and English language learner status, as well as a series of 
background variables specific to this content area. This enables analysis of the achievement data 
based on these factors. As the assessment is administered in years beyond the initial probe, a 
trend line can be established to track changes in student achievement over time. 

Development of the Framework 

The development of the NAEP Technology and Engineering Literacy Framework required the 
participation of a broad array of individuals and organizations to consider the various definitions 
of “technology,” “engineering,” and “technology and engineering literacy” and to devise a single 
framework that takes all of them into account. Among the organizations whose representatives 
provided advice and feedback were the Council of Chief State School Officers, the International 
Society for Technology in Education, the International Technology and Engineering Educators 
Association, the National Academy of Engineering, the Partnership for 21st Century Skills, and 
the State Educational Technology Directors Association. Eighteen outreach meetings were held 
across the country to solicit feedback on drafts of the framework, with approximately 2,000 
participants. Online and paper-based surveys provided specific written feedback from 350 
respondents. 

In determining what students should know and be able to do in the areas of technology and 
engineering and to set forth criteria for the design of the assessment, the framework developers 
drew from a wide variety of sources. These included state, national, and international technology 
and engineering standards; research studies and reports; and a broad array of publications on 
technology and engineering literacy, educational standards, and assessment. To allow for some 
in-depth probing of fundamental knowledge and skills, the framework and the specifications 
represent a distillation rather than a complete representation of the universe of achievement 
outcomes targeted by teaching about technology and engineering. 

Developers of the framework discussed various constraints regarding exactly what can be 
measured. In particular, there are important aspects of technology and engineering literacy that 
are difficult and time-consuming to measure — such as habits of mind, sustained design and 
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research projects, and working as part of a team — but that are valued by engineers, technology 
and engineering educators, and the business community. These can be only partially represented 
in the NAEP Technology and Engineering Literacy Assessment. 

Developing a technology and engineering literacy assessment from this framework will pose a 
number of challenges. For example, because of the current underdeveloped state of assessing 
technology and engineering literacy, there are few existing sample tasks to serve as examples for 
assessment development. 

The framework developers believe this assessment will provide a rich and accurate measure of 
the technology and engineering literacy that students need both for their schooling and for their 
lives. Development of these technology and engineering literacy skills and capabilities is the 
responsibility of all teachers — not only technology and engineering educators but also teachers 
across the curriculum — and also involves the expectations of parents and society. 


xv 



Technology and Engineering Literacy Framework for the 2014 NAEP 


Chapter One: Overview 


Need for a Framework in Technology and Engineering Literacy 

To what extent can young people analyze the pros and cons of a proposal to develop a new 
source of energy? Construct and test a model or prototype? Use the Internet to find and 
summarize data and information to solve a problem or achieve a goal? 

The exploding growth in the world of technology and the need to answer questions similar to 
those above led the National Assessment Governing Board to sponsor the development of a 
framework for a national assessment of technology and engineering literacy. For generations 
students have been taught about technology and have been instructed in the use of various 
technological devices, but there has been no way to know exactly what students understand 
about technologies and their effective use. 

As the framework was being developed, it became clear that the terms “technology,” 
“engineering,” “information communication technology,” “21st-century skills,” and “literacy” 
are defined and used in significantly different ways in formal and informal education, in 
standards, by professional organizations, and in legislation. Therefore, the framework 
development committees recommended a change of the framework title from “technological 
literacy” to “technology and engineering literacy” to encompass general literacy about the use, 
effects, and designing of technologies. The 2014 NAEP Technology and Engineering Literacy 
Framework is a statement about what should be expected of students in terms of their knowledge 
and skills with technology, written to be the basis for an assessment of technology and 
engineering literacy appropriate for all students. It opens the door to seeing what our K-12 
students know about technology and engineering, in the same way that NAEP assesses their 
knowledge and capabilities in reading, mathematics, science, and other subjects. 

Technology and engineering literacy has been defined in a variety of ways, but here it will be 
thought of as the capacity to use, manage, and evaluate the technologies that are most relevant in 
one’s life, including the infonnation and communication technologies that are particularly salient 
in the world today, as well as to understand technological principles and strategies needed to 
develop solutions and achieve goals. 

Because technology is such a crucial component of modem society, it is important that students 
develop an understanding of its range of features and applications, the design process that 
engineers use to develop new technological devices, the trade-offs that must be balanced in 
making decisions about the use of technology, and the way that technology shapes society and 
society shapes technology. Indeed, some have argued that it is time for technology and 
engineering literacy to take its place alongside the traditional literacies in reading, mathematics, 
and science as a set of knowledge and skills that students are expected to develop during their 
years in school. Others go further in conceptualizing “new literacies” in which technologies are 
seen to restructure and support the development of academic and workplace expertise through 
“cyberlearning” in the networked world (National Science Foundation [NSF], 2008). 
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As of 2008, all 50 states were required to report to the U.S. Department of Education on 
technology literacy, using Information and Communication Technology (ICT) standards based 
on the National Education Technology Standards for Students (NETS*S) (International Society 
for Technology in Education [ISTE], 2007). Seven states have formal assessments for 
technology literacy (Metiri Group, 2009). The Standards for Technological Literacy developed 
by the International Technology and Engineering Educators Association (ITEEA) were being 
used in 4 1 states for technology education courses at either the state level or in local school 
districts (Dugger, 2007). A dozen states required technology education for students in at least 
some grades; 22 states offered technology education as an elective. Engineering-based 
technology education may be offered as a separate subject or embedded into other subject areas, 
such as science or social studies. 

Reflecting the increasing importance of the role of technologies in 21st-century life, the National 
Assessment Governing Board decided to develop a framework for a national assessment of 
technology and engineering literacy. In announcing the NAEP Technology and Engineering 
Literacy Assessment, the Governing Board stated that the goals and objectives of the framework 
should be based on the future needs of the nation and individuals, and on the levels of 
technological literacy likely to be expected of students in the first half of the 21st century. 

The primary audience for this framework is the general public. For that reason, the developers 
have attempted to use a minimal amount of technical language and have provided definitions 
when employing terms with which readers may not be familiar. There is also a glossary of terms 
in appendix A. The framework contains sufficient information for policymakers, educators, and 
other interested parties to understand the nature and scope of the assessment. Those interested in 
more technical information can consult the companion document, Assessment and Item 
Specifications for the 2014 NAEP Technology and Engineering Literacy Assessment. 

It is important to note that this framework is an assessment framework, not a curriculum 
framework. That is, although it provides a definition of what students should know and be able to 
do concerning technology and how NAEP will measure that content, it is not intended to tell 
teachers and administrators what should be taught in the classroom, when it should be taught, or 
how it should be taught. 

In addition to laying out the need for a technology and engineering literacy assessment 
framework, this chapter offers background information on NAEP and provides a list of resources 
used to develop the framework, definitions of key terms, a description of the process used for 
developing the framework, an explanation of the challenges of developing a technology and 
engineering literacy assessment framework, and an overview of the framework. 

Background of NAEP 

For more than 35 years, NAEP has measured student achievement nationally, state by state, and 
among 20 large urban districts. NAEP has served as an independent monitor of what students 
know and can do in various subject areas, including reading, mathematics, science, U.S. history, 
and writing. For each subject area measured by NAEP, a framework is used to provide 
recommendations on the content and processes to be assessed, the types of assessment questions 
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to be asked, and the administration of the assessment. Each framework is designed to guide the 
assessment for about a decade until it is updated. 

NAEP results, commonly referred to as The Nation’s Report Card, have become an important 
source of information on what U.S. students know and are able to do in a range of subject areas. 
The resulting data on student knowledge and perfonnance have been accompanied by 
background infonnation that allows analyses of a number of student demographic and 
instructional factors related to achievement. The assessments have been designed to allow 
comparisons of student performance over time and among subgroups of students defined 
according to region, parental education, gender, and race/ethnicity. The Trial Urban District 
Assessment (TUDA) was initiated in 2002 to report on the achievement of public school students 
in large urban districts. Under this program, NAEP has administered its mathematics, reading, 
science, and writing assessments to samples of students in large urban districts that have 
volunteered to participate in NAEP. 

Resources for Planning the Framework 

Any NAEP framework must be guided by NAEP purposes as well as the policies and procedures 
of the Governing Board, which oversees NAEP. For the NAEP Technology and Engineering 
Literacy Assessment, the main purpose of the framework is to establish what students should 
know about and be able to do with technology and to set forth criteria for the design of the 2014 
assessment and future assessments. Meeting this purpose requires a framework built around what 
the communities involved in technology, technology education, educational technology, and 
technology and engineering literacy consider to be the knowledge and skills that are most 
important for NAEP to report. 

In prioritizing the content, the framework developers used the NAEP technological literacy 
steering committee guidelines (summarized later in this chapter). These guidelines recommended 
drawing from the following sources: 

• Existing technology standards from various individual states; 

• National Education Technology Standards by the International Society for Technology in 
Education (ISTE, 2007); 

• Standards for Technological Literacy: Content for the Study of Technology by the 
International Technology and Engineering Educators Association (ITEEA, 2007); 

• Framework for 2 1st Century Learning from the Partnership for 2 1 st Century Skills (P2 1 , 
2007); 

• Influential technology standards from other countries (for example, the United 
Kingdom); 

• Science Framework for the 2009 National Assessment of Educational Progress (National 
Assessment Governing Board, 2008); 

• Benchmarks for Science Literacy by the American Association for the Advancement of 
Science (AAAS, 1993); 

• National Science Education Standards from the National Research Council (NRC, 1996); 


1-3 



Technology and Engineering Literacy Framework for the 2014 NAEP 


• Definitions of technological literacy contained in the No Child Left Behind Act of 2001 
(NCLB, 2001) and the American Recovery and Reinvestment Act of 2009 (ARRA, 

2009); and 

• Research studies and reports, for example. Technically Speaking and Tech Tally: 
Approaches to Assessing Technological Literacy from the National Academy of 
Engineering (NAE) and the National Research Council (NRC); The Intellectual and 
Policy Foundations of the 21st Century Skills Framework (P21, n.d.); and Fostering 
Learning in the Networked World: The Cyberlearning Opportunity and Challenge (NSF, 
2008). 

Tables illustrating how the major assessment areas presented in chapter two are aligned with 
these source documents are presented in appendixes C and D. These sources embody a wealth of 
information about technological literacy and technology education. All of them address similar 
issues of K-12 content and assessment, and in many ways they converge on a broad vision of 
technology and engineering literacy. However, the various documents do not always agree on 
definitions of terms, and in many cases they attach different meanings to phrases such as 
“educational technology” and “technology education,” which a reader outside the field would 
find confusing. Consequently, it is important to establish clear definitions for the purpose of this 
framework and the work of NAEP that will follow. (See the glossary in appendix A for full 
definitions of relevant tenns used in this framework.) 

Definitions of Key Terms Used in This Framework 

Because of the variety of meanings associated with words and terms that are used throughout this 
document, meanings and usages specific to this framework are offered here. The particular 
words and terms at issue are technology, engineering, technology and engineering literacy, and 
educational technology. It is important to note that these definitions have been developed for the 
sole purpose of informing NAEP about the field of technology and engineering literacy. No 
additional claim is made regarding the usefulness of these definitions for other purposes — and, in 
particular, they should not be used for the interpretation of state and local assessments, since 
these may be based on different definitions of technology and engineering literacy. This 
framework defines technology, engineering, and technology and engineering literacy as: 


Technology is any modification of the natural or designed world done to fulfill human needs 
or desires. 

Engineering is a systematic and often iterative approach to designing objects, processes, and 
systems to meet human needs and wants. 

Technology and engineering literacy is the capacity to use, understand, and evaluate 
technology as well as to understand technological principles and strategies needed to develop 
solutions and achieve goals. For purposes of this framework, it comprises three areas: 
Technology and Society, Design and Systems, and Infonnation and Communication 
Technology. 
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The following sections expand and elaborate on the above brief definitions of “technology,” 
“engineering,” and “technology and engineering literacy” with the goal of describing the 
knowledge and capabilities that are essential for citizens in the 21st century — and, in particular, 
in a way that can be assessed through an on-demand, large-scale assessment. 

Technology 

Research shows that most Americans associate technology mainly with computers and related 
electronic devices (Cunningham, Lachapelle, & Lindgren-Streicher, 2005). However, while the 
computer is certainly an important example of technology — and one that plays an especially 
important role in this framework — the term “technology” has a broader and deeper meaning. 
Broadly speaking, technology is any modification of the natural or designed world done to fulfill 
human needs or desires. This definition sees technology as encompassing the entire human-made 
world, from the simplest artifacts, such as paper and pencil or a wooden flute, to the most 
complex, including the Internet, medical imaging devices, and a country’s entire transportation 
system. Technology helps people do their jobs: giant particle colliders for physicists, oil paints 
and canvases for artists, and email for just about everyone. Technology also makes leisure time 
more enjoyable: movies, music, and electronic games to entertain, automobiles to get people 
where they want to go, audio and imaging tools to support creative expression, and books (either 
e-books or the traditional paper version) to tell stories of other places and times. 

But technology is not just the products or things that people create. It includes the entire 
infrastructure needed to support the processes used to design, manufacture, operate, and repair 
technological artifacts, from corporate headquarters and engineering schools to manufacturing 
plants, media outlets, and distribution networks (Shakrani & Pearson, 2008). Technology also 
includes the cyberinfrastructure and participatory technologies that open up greater access to 
complex learning and connections of experiences across settings (NSF, 2008). 

Throughout history, technology has been one of the major factors shaping human life and human 
civilization, and, indeed, major periods of human development have typically been identified by 
the dominant technologies of the period: Stone Age, Bronze Age, Iron Age, Industrial Age, and, 
today, the Infonnation Age. Technology itself is constantly changing and evolving, as are its 
effects on society. Ten thousand years ago humans took the first steps toward agriculture with 
the purposeful planting of seeds; 100 years ago, farmers and plant scientists were regularly 
improving crops through hybridization; today, genetic engineering has been harnessed to create 
specially designed crops and farm animals. Perhaps the most dramatic example of technological 
evolution from today’s perspective is the rapid development of communications technology, 
from the invention of the telegraph and telephone in the 19th century to the development of 
radio, television, and the Internet in the 20th century — and to the past decade’s explosion of 
electronic communication, and social networking (Solomon & Schrum, 2007). Technology has 
become an enabling force behind globalization, knowledge work, and entrepreneurship (Metiri 
Group, 2006). With each of these changes comes new capabilities — and new challenges. 
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Engineering 

Engineering is the process of designing the human-made world. The process typically begins 
with the specifications of needs or wants. Engineers identify constraints, analyze the features of 
systems, and devise plans for developing solutions. Solutions may take the form of artifacts such 
as computer chips or bridges. Solutions may also take the fonn of improved processes such as 
assembly lines or traffic control. Engineering processes are typically iterative, involving testing 
and revisions. Engineers understand the nature of the technology area to be modified, engage in 
systems thinking, work through engineering design processes, and conduct maintenance and 
troubleshooting. 

Technology and Engineering Literacy 

Having defined “technology” and “engineering” broadly in these ways, “technology and 
engineering literacy” can be defined in an equally broad fashion as the capacity to use, 
understand, and evaluate technology as well as to understand technological principles and 
strategies needed to develop solutions and achieve goals. Technology and engineering literacy, 
like scientific, mathematical, or language literacy, is a measure of how well individuals have 
mastered the processes and tools they need to participate intelligently and thoughtfully in the 
world around them. As described in reports from the National Academy of Engineering (NAE, 
2006), the International Society for Technology in Education (ISTE, 2007), and the International 
Technology and Engineering Educators Association (ITEEA, 2007), technological literacy 
includes knowledge, capabilities, and critical thinking and decision-making skills (McAnear, 
2009). From these documents, lists can be extracted of what a person who is literate in 
technology and engineering should know and be able to do. 

Figure 1 depicts the major assessment areas and practices of the NAEP Technology and 
Engineering Literacy Framework. At the center, in blue, are the three areas of technology and 
engineering literacy that will be assessed by NAEP: Technology and Society, Design and 
Systems, and Information and Communication Technology (ICT). The surrounding yellow circle 
represents three overarching types of thinking and reasoning that generalize across the major 
assessment areas. These crosscutting practices will apply across tasks and individual test 
questions, also referred to as items, in the areas of Technology and Society, Design and Systems, 
and ICT. These key components of the 2014 NAEP Technology and Engineering Literacy 
Framework are described on the next page. 
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Three Areas of Technology and Engineering Literacy 


In recent decades, students in the United States 
have experienced three quite different (though 
by no means inconsistent) approaches to 
technology and engineering literacy. These are 
the science, technology, and society approach; 
the technology education approach; and the 
information and communication technology 
approach. In recognition of the importance, 
educational value, and interdependence of 
these three approaches, this framework 
includes all three under its broad definition of 
technology and engineering literacy. In 
recognition of the distinct goals and teaching 
methods involved in each, this framework 
recommends that assessment results be 
reported for each of these areas to monitor and 
analyze the results of each approach over time. 

The next few paragraphs present a brief 
description of each of these approaches. 

• The Technology and Society assessment area has its roots in the science, technology, and 
society (STS) approach. In 1990 the board of directors of the National Science Teachers 
Association defined STS as the “teaching and learning of science and technology in the context 
of human experience” (NSTA, 2006, pp. 229-230). In practice many STS programs use societal 
issues as course organizers, including space travel, insecticide use, nutrition, disease, ozone, 
global warming, and other concerns reported in the popular press. Since technological advances 
and decisions lie at the core of such issues, the focus in discussing them is often on the 
technology involved (Yager & Akcay, 2008). A survey of engineering and technology in state 
science standards found that a majority of state standards reflect the STS approach (Koehler, 
Faraclas, Giblin, Kazerounian, & Moss, 2006). The STS approach is represented in this 
framework under “Technology and Society.” 

• The Design and Systems assessment area is partly rooted in the school subject kn own as 
industrial arts (Dugger, 2005), a popular subject area throughout most of the 20th century, which 
provided education in the use of hand and power tools for fabricating objects from wood, metal, 
or other materials, as well as instruction on industrial processes. As conceived today by the 
field’s professional organization, the International Technology and Engineering Educators 
Association (ITEEA), technological literacy “involves a vision where each citizen has a degree 
of knowledge about the nature, behavior, power, and consequences of technology from a broad 
perspective. Inherently, it involves educational programs where learners become engaged in 
critical thinking as they design and develop products, systems, and environments to solve 
practical problems” (ITEEA, 1996, p. 1). Goals in technology education include creating a broad 
understanding of technology and engineering as well as developing specific capabilities in both 
areas (ITEEA, 2007). A survey of state science standards (Koehler et al, 2006) found that many 



Figure 1 - Major Assessment Areas and Practices 
of Technology and Engineering Literacy 
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states, especially those in the Northeast, include standards consistent with this approach, 
although not as many as those whose standards relate to STS. The engineering design approach 
is represented in this framework under the heading “Design and Systems.” 

• Infonnation and communication technology (ICT) is a third approach that has been growing in 
importance over the past three decades. The pervasiveness of technology in school, home, work, 
and play has profound implications for learning in schools and throughout life (p. 21, n.d.). The 
field’s major professional organization, the International Society for Technology in Education 
(ISTE), was formed in 1989 by the merger of two associations concerned primarily with the use 
of computers in education. Today, the vision of ICT is much broader than the use of computers 
alone, having expanded from the earlier vision of technology as a teaching tool to today’s 
philosophy of technology as a learning tool. That is, the focus is no longer on using technology 
to assist teachers but rather on giving students new and more powerful ways to gather and assess 
information, think creatively, solve problems, and communicate. As expressed in the society’s 
National Educational Technology Standards (ISTE, 2007), ICT includes a variety of student 
skills that overlap with other areas, such as creativity and innovation; communication and 
collaboration; research and information fluency; and critical thinking, problem-solving, and 
decision-making. These skills are applied specifically to the use of digital technologies and 
media, including the Internet and other networking applications. 

Although these infonnation technologies make up one component of technology, broadly 
defined, they have been responsible for many of the most profound changes that have taken place 
in society over the past several decades. And the variety, uses, and power of such infonnation 
tools are expected to grow rapidly over the next decade (The New Media Consortium, 2009). 
Media, telecommunication, and networked technologies are evolving into powerful support 
systems for acquiring skills needed in the 21st century (p. 21, n.d.). 

“Every young person will need to use ICT in many different ways in their adult lives, in 
order to participate fully in a modem society” (Organisation for Economic Co-operation and 
Development [OECD], 2006). The ICT approach to technology and engineering literacy is 
represented in this framework under the heading “Infonnation and Communication Technology.” 

Three Practices 

A person who is literate in technology and engineering should be able to apply “crosscutting 
practices,” or generalizable ways of thinking, reasoning, and acting that are important across all 
areas of technology and engineering literacy. As depicted in figure 1, these practices are 
employed within and across the 3 major assessment areas. The practices can be grouped into 
these three broad categories, with several examples of each type of practice: 

Understanding Technological Principles 

• Understands the nature of technology in its broadest sense 

• Is aware of the various digital tools and their appropriateness for different tasks 

• Knows how technology is created and how it shapes society and in turn is shaped by 

society 
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• Understands basic engineering concepts and terms, such as systems, constraints, and 
trade-offs 

• Understands how cultural differences can affect technological choices 
Developing Solutions and Achieving Goals 

• Demonstrates appropriate usage of a wide range of technological tools and systems, 
ranging from kitchen appliances and alann clocks to cars, computers, communication 
devices, and the Internet 

• Can apply technological concepts and skills creatively, including those of engineering 
design and infonnation technology, to solve problems and meet goals 

• Collects and analyzes data to develop a solution and complete a project 

• Uses multiple processes and diverse perspectives to explore alternative solutions 

• Can evaluate claims and make intelligent decisions 

Communicating and Collaborating 

• Communicates infonnation and ideas effectively to multiple audiences using a variety of 
media and fonnats 

• Participates thoughtfully and productively in discussing critical societal issues involving 
technology related to humans, the environment, knowledge, and citizenship 

• Collaborates with peers and experts 


The framework recommends that results of the NAEP Technology and Engineering Literacy 
Assessment be reported separately for the three major assessment areas of Technology and 
Society, Design and Systems, and ICT, although it cannot be stressed strongly enough that 
today’s youth are expected to acquire knowledge and skills in all three areas of technology and 
engineering literacy. These areas are neither learned separately nor applied separately; they 
overlap and interact. A person who is literate in technology and engineering understands and is 
able to analyze the relationship between technology and society, has a broad understanding of 
technology and can solve problems using the engineering design process, and is able to make 
fluent use of digital technologies and media in creative and innovative ways. Specific assessment 
targets related to the three areas are described at length in chapter two. 


Educational Technology 

Although it is not an assessment target for the purposes of NAEP, the field of educational 
technology provides another example of a common use of the term “technology.” Broadly 
speaking, the field of educational technology is concerned with the use of various types of 
equipment as teaching and learning aids. Many teachers remember when overhead projectors 
were in widespread use or when whiteboards replaced chalkboards. Advocacy for the use of 
computers in classrooms began more than 20 years ago, and the uses of computers have evolved 
rapidly from computers-as-teachers to computers-as-leaming-tools. Today a vast array of 
computer applications is available for use in all school subjects, and these applications are 
fundamentally altering the way students learn in school, giving them unprecedented input into 
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and control of their own learning. Some devices, such as interactive whiteboards, combine 
technologies for entirely new purposes. An area of digital or cyber literacy is emerging that 
encompasses newer forms of technology and media (Kress, 2003; Livingstone, van Couvering, 
and Thumin, 2008). Traditions of media and information literacy are converging and focusing on 
skills needed to take advantage of digital systems for representing and distributing information 
(Livingstone, 2002). The variety and use of such tools for learning, expression, and 
communication are expected to expand rapidly over the next decade, affecting the way all 
people — not just students — work, collaborate, and communicate (The New Media Consortium, 
2009). The 2014 NAEP Technology and Engineering Literacy Assessment will take advantage 
of new developments in educational technology as one of the first NAEP assessments to be 
administered entirely by computer. 

The Relationship Among Science, Technology, Engineering, and Mathematics 

Science, technology, engineering, and mathematics are so closely interlinked that it is often 
difficult to know where one starts and the other ends. Students in science classes are often taught 
about technology, engineering, and mathematics, while students in technology classes leam 
about science, engineering, and mathematics. Technologies are changing fundamentally the ways 
scientists work, and are becoming important components of science education. Students’ skills in 
using the tools of science are becoming components of the “new literacies” (Quellmalz and 
Haertel, 2008). In a recent report on cyber learning, the National Science Foundation points out 
that research has demonstrated that “incorporating information and communications technology 
into science and mathematics can restructure the necessary expertise for reasoning and learning 
in these domains, in effect opening up greater access to complex subject matter.” Examples 
include multiple linked representations in mathematics and modeling and visualizations for 
understanding and investigating complex science (NSF, 2008, p. 13). 

For the purposes of designing a framework to assess technology and engineering literacy, it is 
important to keep the distinctions among the science, technology, and engineering clear. The 
relationship among engineering, science, and technology is explained this way in the joint 
National Academy of Engineering/National Research Council publication Technically Speaking : 

Science and technology are tightly coupled. A scientific understanding of the 
natural world is the basis for much of technological development today. The design 
of computer chips, for instance, depends on a detailed understanding of the 
electrical properties of silicon and other materials. The design of a drug to fight a 
specific disease is made possible by knowledge of how proteins and other 
biological molecules are structured and how they interact. 

Conversely, technology is the basis for a good part of scientific research. The 
climate models meteorologists use to study global warming require supercomputers 
to run the simulations. And like most of us, scientists in all fields depend on the 
telephone, the Internet, and jet travel (NAE & NRC, 2002, pp. 13-14). 

One other distinction that is important to make is between technology and engineering. Again the 
explanation from Technically Speaking is helpful: 
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Technology is a product of engineering and science, the study of the natural 
world. Science has two parts: (1) a body of knowledge that has been accumulated 
over time, and (2) a process — scientific inquiry — that generates knowledge about 
the natural world. Engineering, too, consists of a body of knowledge — in this 
case, knowledge of the design and creation of human-made products — and a 
process for solving problems (NAE & NRC, 2002, p. 13). 

Of the three terms — science, technology, and engineering — the clearest parallel is between 
science and engineering, as both represent an approach to knowledge taken by a group of well- 
trained professionals. As explained in the National Science Education Standards (NRC, 1996, p. 
166), “ Scientists propose explanations for questions about the natural world, and engineers 
propose solutions relating to human problems, needs, and aspirations.” 

A fourth area that is often associated with these other three is mathematics. Although 
mathematics is a field in its own right, distinct from science and engineering, mathematical tools 
are essential to the work of both scientists and engineers. In fact, science, technology, 
engineering, and mathematics are so intimately connected that they are frequently referred to by 
the acronym STEM. 

The Framework Development Process 

In October 2008, the Governing Board awarded a contract to WestEd to develop a framework 
and specifications for assessing technological literacy. In carrying out its work, WestEd 
collaborated with the Council of Chief State School Officers (CCSSO), ISTE, ITEEA, the 
Partnership for 21st Century Skills, and the State Educational Technology Directors Association 
(SETDA). In working with these groups, WestEd used a process for developing the framework 
and related products that was inclusive, deliberate, and designed to achieve as much broad-based 
input as possible. 

A two-tiered committee structure with a steering committee and a planning committee provided 
the expertise to develop the framework as specified by the Governing Board. (See “NAEP 
Technology and Engineering Literacy Project Committees and Staff” section for lists of 
committee members.) The two committees were composed of members who were diverse in 
tenns of role, gender, race or ethnicity, region of the country, perspective, and expertise 
regarding the content of the assessment to be developed. 

The steering committee members included leaders from a variety of fields and subject areas, 
including schools, engineering, education, 21st-century skills, the Internet, business, science 
education, general education, and assessment. The co-chairs were balanced, with one 
representing technology in schools, and the other, STEM and assessment. Functioning as a 
policy and oversight body, this group outlined the planning committee’s responsibilities in 
developing the framework. The committee reviewed and provided feedback on drafts of the 
framework and related materials. The interaction between the two committees was iterative over 
the course of the project. 
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The planning committee, as supported by the project staff, was the development and production 
group responsible for drafting the framework, specifications, recommendations for background 
variables, and preliminary technology and engineering literacy achievement level definitions. 
This committee was composed of business leaders, researchers, state and district technology 
coordinators, teachers, and representatives from educational organizations as well as experts in 
research, assessment, and evaluation. As with the steering committee, the planning committee 

Summary of the Steering Committee Guidelines 

• The following definition of technology and engineering literacy should be used: 
Technology and engineering literacy is the capacity to use, understand, and evaluate 
technology as well as to understand technological principles and strategies needed to 
develop solutions and achieve goals. The steering committee also suggested that the 
planning committee examine other definitions of technology and engineering literacy, 
especially those used by the states. 

• The assessment should consist of technological content areas, to be reported as scale 
scores, and technological practices that characterize the field. The content areas must 
include Design and Systems, Infonnation and Communication Technology (ICT), and 
Technology and Society. 

• Three practices should be addressed in the assessment: Understanding Technological 
Principles, Developing Solutions and Achieving Goals, and Communicating and 
Collaborating. 

• The content and context for the assessment should be infonned by state standards and 
assessments, national and international standards, and research. 

• Assessment tasks and items should minimize the need for prior knowledge of specific 
technologies other than ICT; tasks should relate to real-world problems and contexts. 

• As far as possible, life situations familiar to students and local and contemporaneous 
conditions should be used to design tasks as a way to confer relevance to each grade level 
being assessed. 

• The assessment should focus on a broad base of knowledge and skills, not on specific 
technological processes that may change. 

• The assessment should use innovative computer-based strategies that are informed by 
research on learning and are related to the assessment targets. 

• The computer-based assessment strategies should be informed by what is known about all 
learners, including English language learners (ELL) and students with disabilities. 

• All the items and tasks included as exemplars in the framework and specifications 
documents should have feasible responses that can be awarded score points. 
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co-chairs were balanced, with one being an expert in ICT-based learning and assessment and the 
other being an expert in K-12 science and engineering education. 

The planning committee’s work was guided by policies, goals, and principles identified by the 
steering committee (see summary box on the previous page; for full text, see appendix B). In 
addition to the sources cited previously, the planning committee relied on guidance provided by 
NAEP Technological Literacy Framework and Specifications Development: Issues and 
Recommendations, a paper prepared by Sharif M. Shakrani and Greg Pearson for the Governing 
Board. 

The work was carried out in a series of meetings, with numerous telephone calls and email 
exchanges between meetings. From December 2008 through September 2009, the steering 
committee met four times and the planning committee met six times. Three steering committee 
and planning committee meetings overlapped so that the two committees could share 
understandings and discuss critical issues. Governing Board staff supported and participated in 
all of the meetings. In addition, between formal work sessions, Governing Board members and 
staff provided feedback and guidance on project documents and processes. 

After the development of initial drafts of the framework, WestEd led a series of outreach efforts 
to solicit feedback. Formal activities included, but were not limited to, presentations and sessions 
with industry representatives (for example, IBM and Cisco), ITEEA, the Organisation for 
Economic Co-Operation and Development (OECD) International ICT Research Workshop, 
CCSSO, SETDA, ISTE, and the Partnership for 21st Century Skills Webinar. (See NAEP 
Technology and Engineering Literacy Project Committees and Staff section for more complete 
lists of individuals and organizations that contributed to the development of this framework.) The 
planning committee reviewed feedback from these groups as well as from the steering committee 
and made revisions, as it deemed appropriate. Governing Board members and staff closely 
monitored the project and provided comments on draft documents. After final approval from the 
steering committee, the framework was submitted to the Governing Board for action. 

Challenges of Developing the NAEP Technology and Engineering Literacy 
Framework 

There were a number of challenges in developing the 2014 NAEP Technology and Engineering 
Literacy Framework that were not necessarily encountered in developing other NAEP 
frameworks. These included: (1) the newness of the endeavor, (2) diffuse curricula, (3) varying 
definitions, (4) measurement constraints, (5) time and resource constraints, (6) designing an 
entirely computer-based assessment, and (7) predicting future changes in technology. Each of 
these challenges is discussed below. 

Newness of the Endeavor 

Technology and engineering literacy is a growing and evolving area. Unlike other NAEP 
subjects, such as reading or mathematics, there is no existing NAEP framework to draw on. 
Moreover, the existing item banks in the United States and other countries are very limited 
(NAE, 2006). The technology and engineering literacy staff and committee members obtained 
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only a limited number of sample items from outside sources, reflecting the immature state of 
assessing technology and engineering literacy. 

Diffuse Curriculum 

Unlike science and mathematics, which have a sequential curriculum taught by subject- area 
specialists in high school or by generalists in elementary school, technology and engineering 
education as a whole does not have a unified scope and sequence. Some individual courses (for 
example, science, technology, and society; pre-engineering; and computer modeling) are likely 
to follow state standards and have a specified curriculum with a scope and sequence, but these 
individual courses are generally not grouped together under the rubric of technology courses. 
When they are, the courses under such a heading may vary from place to place. ICT has also 
been integrated into the curriculum in a variety of ways. While ICT learning is often infused into 
existing core subjects, it is not always assessed and reported as part of these subjects. In addition, 
there is not a clear scope and sequence for ICT knowledge and skills, either as a stand-alone 
curriculum or integrated into core subjects, which may result in an inconsistent application of 
technology literacy standards across different grades, subjects, and states. As mentioned earlier, 
all teachers have a role in teaching technology, so in most cases, teachers are not singled out as 
technology teachers in the same way that, for example, mathematics or history teachers are 
identified with those subject areas. The result (and implication for this assessment) is that the 
specific technology concepts and practices to which students have been exposed are hit-and-miss 
and mostly unknown. Students can say what mathematics or science courses they have taken, but 
specifying the range of their education in technology and engineering and their use is more 
ambiguous. 

Varying Definitions 

One of the most debated issues in developing the framework was the definition of “technology 
and engineering literacy,” as different definitions abound. Indeed, even the terms are not agreed 
upon, as some organizations refer to “technological literacy” while others refer to “technology 
literacy” or “engineering literacy.” In this report, for consistency, the term “technology and 
engineering literacy” will be used throughout while recognizing that this terminology differs 
from what is used by some groups. ITEEA, the National Research Council, and ISTE have 
definitions of technology and engineering literacy. Meanwhile, the federal No Child Left Behind 
Act of 2001 (NCLB) required that every student be “technologically literate by the time the 
student finishes the eighth grade,” but the law itself is vague in defining what technological 
literacy is. States have therefore had flexibility in determining what technology and engineering 
literacy means and how it should be assessed. 

Many states have adopted a common definition worked out by SETDA in 2002, which states, 
“Technology literacy is the ability to responsibly use appropriate technology to communicate, 
solve problems, and access, manage, integrate, evaluate, and create information to improve 
learning in all subject areas and to acquire lifelong knowledge and skills in the 21st century.” 

The federal American Recovery and Reinvestment Act of 2009 adds real-world consequences to 
this shared definition by providing grants to state and local agencies and schools based on their 
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abilities to meet goals defined by long-range educational technology plans, most of which 
include this definition. 

The NAEP Technology and Engineering Literacy Framework attempts to unify the concepts and 
skills presented in these and other definitions under one umbrella definition. The definition 
presented earlier in this chapter is used only to understand the results of the NAEP assessment in 
technology and engineering literacy. While this framework and subsequent NAEP results may be 
informative to education administrators, policyholders, industry and business leaders, and the 
general public, the definition of technology and engineering literacy presented here should not be 
used to interpret results from other assessments used at state and local levels. To further 
distinguish the 2014 NAEP Technology and Engineering Literacy Assessment from other 
technology and engineering literacy assessments, the project committees have recommended that 
the results be reported in terms of three individual scores, each reflecting performance in one of 
the three main areas of technology and engineering literacy: Technology and Society, Design and 
Systems, and Information and Communication Technology. An overall composite score will also 
be reported. 

Measurement Constraints 

NAEP, like any large-scale assessment in education, the workplace, or clinical practice, is 
constrained in what it can measure. This has implications for the proper interpretation of NAEP 
Technology and Engineering Literacy Assessment results. The framework is an assessment 
framework, not a curriculum framework. Although the two are clearly related, each has a 
different purpose and a different set of underlying assumptions. A curriculum framework is 
designed to infonn instruction, to guide what is taught, and, often, to guide how it is taught. It 
represents a wide universe of learning outcomes from which educators pick and choose what and 
how they teach. An assessment framework is a subset of the achievement universe from which 
assessment developers must choose to develop sets of items that can be assessed within time and 
resource constraints. Hence, the content to be assessed by NAEP has been identified as that 
considered central to technology and engineering literacy. 

As a result, some important outcomes of technology and engineering literacy (broadly defined) 
that are valued by general educators, engineers, teachers of technology, and the business 
community but that are difficult and time-consuming to measure — such as habits of mind, 
sustained projects, and collaboration — will be only partially represented in the framework and on 
the NAEP Technology and Engineering Literacy Assessment. Moreover, the wide range of 
technology and engineering standards in the guiding national documents that were incorporated 
into the framework had to be reduced in number to allow some in-depth probing of fundamental 
knowledge and skills. As a result, the framework and the specifications represent a distillation 
rather than a complete representation of the original universe of achievement outcomes specified 
by technology and engineering education documents. 
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Time and Resource Constraints 

Time and resources limit what NAEP can assess. Like most standardized assessments, NAEP is 
an “on-demand” assessment. That is, it is given as a scheduled event outside the normal 
classroom routine with uniform conditions for all of the students being assessed. In particular, 
NAEP has a limited amount of time — in this case, approximately one hour per student — to 
ascertain what students know and can do. However, standards presented by professional 
associations and the states contain goals that require an extended amount of time (days, weeks, 
or months) to assess. To assess the achievement of students in the kinds of extended activities 
that are a central feature of these other standards and of many curricula, it would be necessary to 
know a number of things about the students, including their: 

• Reasoning while framing their goals; 

• Planning for projects and the implementation of the plan; 

• Skills in using technologies to gather, manage, and analyze data and information related 
to project goals; 

• Capabilities to meet unpredictable challenges that arise during actual, ongoing problem- 
solving and achievement of goals; 

• Lines of argument in deciding how to alter their approaches in the light of new evidence; 

• Engagement with peers and experts in addressing goals and deciding how to achieve 
them; and 

• Deliberations and reasoning when evaluating progress, trade-offs, and results. 

NAEP, like other on-demand assessments, then, cannot be used to draw conclusions about 
student achievement with respect to the full range of goals of technology education, broadly 
defined. States, districts, schools, and teachers can supplement NAEP and other standardized 
assessments to assess the full range of education standards that address technology and 
engineering literacy. In addition to describing the content and format of an examination, 
assessment frameworks, like this one, signal to the public and to teachers some core elements of 
a subject that are important. 

Designing a Computer-Based Assessment 

Although some NAEP assessments (the 2009 Science Assessment, for example) have called for 
interactive computer tasks, so far only the NAEP Writing Assessment has been totally computer- 
based. The design challenges of creating such an assessment include: 

• Developing the requisite number of tasks and items (test questions), especially since so 
few tasks and items exist that can serve as samples. 

• Constructing tasks and items that provide whatever prior knowledge is required to answer 
the question. As so many contexts are available in which to set items, developers cannot 
assume that students will have prior knowledge of the specific topics (for example, core 
subjects such as the humanities or mathematics) or technologies (for example, 
transportation, health, or electronics) within the context. Items must not require students 
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to have prior knowledge of specific technologies, and the knowledge required about 
particular technologies must be presented in the item. 

• Determining the features and functions of the complete tools students will use. 

• Determining what aspects of student responses to an item need to be assessed. Are the 
attempts a student makes while trying out a design or using a simulation important to 
capture? What about the pathway the student follows or the number of mistakes made 
before getting a correct answer? Rather than a single question and answer, an item might 
have several components that are assessed. 

In addition to the issues above, there will also be administrative challenges, such as whose 
computers the students use to complete the assessment, handling students’ different levels of 
access to computer technology, and contingencies in case equipment malfunctions. The 
framework designers were aware of these factors when developing the framework, but they 
focused on the design factors, leaving the challenge of detennining how best to administer the 
NAEP Technology and Engineering Literacy Assessment to those involved in the assessment 
development phase. 

Predicting Future Changes in Technology 

The framework attempts to strike a balance between what can reasonably be predicted about 
future technology and engineering literacy education and what students are likely to encounter in 
their curriculum and instruction now and over the next decade. For example, specific 
communication technologies in use today (Internet-connected multimedia smartphones and 
personal digital assistants [PDAs]) would not have been familiar to students a decade ago and 
may well be obsolete a decade from now. 

The framework is intended to be both forward-looking (in terms of what technology content and 
usage will be of central importance in the future) and reflective (in terms of current technology). 
Because it is impossible to predict with certainty the shape of educational technology and 
technology education, the choices made for 2014 should be revisited in response to future 
developments. 

It is a significant challenge to write a framework for the future, and the challenge is especially 
great for the subject of technology and engineering literacy. 

Overview of Framework Chapters 

The following text describes the content of the remaining four chapters of the 2014 NAEP 
Technology and Engineering Literacy Framework. 

Chapter Two: Areas of Technology and Engineering Literacy 

This core chapter identifies the assessment targets for the 2014 assessment of technology and 
engineering literacy. The targets are grouped into the three major areas: Technology and Society, 
Design and Systems, and Information and Communication Technology (ICT). Each of the major 


1-17 



Technology and Engineering Literacy Framework for the 2014 NAEP 


areas is broken down into subareas. Each subarea has a listing of key principles and a chart 
identifying what students should kn ow and be able to do at grades 4, 8, and 12. Each subarea has 
between 6 and 15 assessment targets. 

Chapter Three: Practices and Contexts for Technology and Engineering Literacy 

Chapter three has two major purposes. First, it articulates the kinds of thinking and reasoning 
that students are expected to demonstrate when responding to the assessment tasks and items. 
Three practices are presented: (1) understanding technological principles, (2) developing 
solutions and achieving goals, and (3) communicating and collaborating. Each practice is applied 
to the three major technology and engineering literacy areas in a chart. There are also tables that 
apply the practices to selected principles and to the subareas. Illustrative tasks and items suggest 
how the practices can be represented along with targets in each of the areas and subareas. 

Next, the chapter describes the contexts for the assessment — that is, the core school subjects and 
areas of technology, such as humanities, social sciences, medical imaging, publishing, or 
recycling, that will serve as backdrops for assessment questions. The choice and presentation of 
contexts are important because the framework cannot assume that students have prior knowledge 
of a specific topic or technology. If infonnation about a specific type of technology is needed to 
respond to an item, this infonnation must be provided to students in the item as contextual detail. 
Potential contexts are discussed within the three major areas: Technology and Society, Design 
and Systems, and Information and Communication Technology. Each of the three areas has a 
chart that provides examples of how different contexts can be used to fonnulate tasks and items 
in each of the three assessment areas. 

Chapter Four: Overview of the Assessment Design 

This chapter is an overview of the major components of the NAEP Technology and Engineering 
Literacy Assessment. It discusses the types of tasks and items to which students will respond. 
These will be scenario-based items and discrete items. Examples illustrate how students might 
respond and what parts of their response would be assessed. Balance in four ways is an important 
requirement in the assessment: balance by major assessment areas and grades, balance by 
technological practices and grades, balance by assessment set types and grades, and balance by 
response types and grades. The chapter has charts containing percentages of testing time 
recommended for each of these features. The chapter concludes with suggestions for universal 
design and adaptations for students with disabilities, and English language learners. 

Chapter Five: Reporting Results of the NAEP Technology and Engineering Literacy 
Assessment 

The last chapter of the framework discusses the issues involved in reporting data from the 
Technology and Engineering Literacy Assessment. Topics include NAEP sampling techniques, 
how NAEP results are reported, reporting scale scores and achievement levels, and reporting 
background variables. Appropriate and inappropriate uses of NAEP reporting are also discussed, 
especially in light of the varying definitions of technology and engineering literacy. 
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Chapter Two: Areas of Technology and Engineering 
Literacy 


Introduction 

This chapter describes the essential knowledge and capabilities that will be assessed on the 
NAEP Technology and Engineering Literacy Assessment beginning in 2014. Although it is not 
possible to assess every aspect of technology and engineering literacy, this framework identifies 
a set of assessment targets related to the nature, processes, and uses of technology and 
engineering that are essential for 21st-century citizens. The assessment targets are organized into 
three major areas: Technology and Society, Design and Systems, and Information and 
Communication Technology. 


These three areas of technology and 
engineering literacy are interconnected. The 
relationship among these three major 
assessment areas can be illustrated as a three- 
sided pyramid in which each side supports the 
other two. For example, to address an issue 
related to technology and society, such as 
clean water, energy needs, or information 
research, a person who is literate in technology 
and engineering must understand technological 
systems and the engineering design process 
and be able to use various infonnation and 
communication technologies to research the 
problem and develop possible solutions. 



This chapter provides descriptions of each of the major areas of technology and engineering 
literacy as well as subareas and tables of assessment targets that specify what students in grades 
4, 8, and 12 should know and be able to do. The assessment targets describing what students 
should be able to do foreshadow the crosscutting “practices” — ways of thinking and reasoning — 
described in chapter three. 

It will be apparent when reading assessment targets across the grade-level rows that learning is 
cumulative — that is, later grades build on what has been learned in earlier grades, so that 
students develop a greater sophistication and depth of understanding as they advance in school. 
For instance, elementary school students think of a technological change in terms of a succession 
of “products,” as in the evolution of writing technology from clay tablets to pen and paper and to 
computers and printers. Middle school students are able to think in terms of technological 
“processes,” such as the processing of food, or the extraction of metal from ore. High school 
students have learned to think in terms of technological “systems,” such as a city’s public 
transportation system or water purification system. 
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There is some overlap among the three major assessment areas. For example, there may be 
references to Information and Communication Technology (ICT) and to Design and Systems 
within the Technology and Society area. This is due in part to the mutual support that these 
technological principles and skills lend to each other, and it serves to emphasize that individuals 
who are literate in technology and engineering can bring these ways of thinking and acting to 
bear on any problem or goal. Thus, ICT knowledge and skills are called on during the design of 
technologies; information and communication technologies are developed by engineering design 
processes; and the myriad technologies designed to meet human needs, including the ubiquitous 
information and communication technologies, are influenced by and have impacts on society. 

The 3 major assessment areas of technology and engineering literacy and their corresponding 
subareas are presented in table 2.1 and briefly summarized below. 

Table 2.1 Major areas and subareas of 2014 NAEP Technology and Engineering Literacy 
Assessment 


Technology and Society 

Design and Systems 

Information and 
Communication Technology 
(ICT) 

A. Interaction of Technology 
and Humans 

B. Effects of Technology on 
the Natural World 

C. Effects of Technology on 
the World of Information 
and Knowledge 

D. Ethics, Equity, and 
Responsibility 

A. Nature of Technology 

B. Engineering Design 

C. Systems Thinking 

D. Maintenance and 
Troubleshooting 

A. Construction and 
Exchange of Ideas and 
Solutions 

B. Information Research 

C. Investigation of Problems 

D. Acknowledgment of Ideas 
and Information 

E. Selection and Use of 
Digital Tools 


Technology and Society 

A. Interaction of Technology and Humans concerns the ways society drives the improvement 
and creation of new technologies, and how technologies serve society as well as change it. 

B. Effects of Technology on the Natural World is about the positive and negative ways that 
technologies affect the natural world. 

C. Effects of Technology on the World of Information and Knowledge focuses on the rapidly 
expanding and changing ways that information and communications technologies enable data to 
be stored, organized, and accessed and on how those changes bring about benefits and challenges 
for society. 

D. Ethics, Equity, and Responsibility concerns the profound effects that technologies have on 
people, how those effects can widen or narrow disparities, and the responsibility that people 
have for the societal consequences of their technological decisions. 
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Design and Systems 

A. Nature of Technology offers a broad definition of technology as consisting of all the 
products, processes, and systems created by people to meet human needs and desires. 

B. Engineering Design is a systematic approach to creating solutions to technological problems 
and finding ways to meet people’s needs and desires. 

C. Systems Thinking is a way of thinking about devices and situations so as to better understand 
interactions among components, root causes of problems, and the consequences of various 
solutions. 

D. Maintenance and Troubleshooting is the set of methods used to prevent technological 
devices and systems from breaking down and to diagnose and fix them when they fail. 

Information and Communication Technology (ICT) 

A. Construction and Exchange of Ideas and Solutions concerns an essential set of skills 
needed for using Infonnation and Communication Technology (ICT) and media to communicate 
ideas and collaborate with others. 

B. Information Research includes the capability to employ technologies and media to find, 
evaluate, analyze, organize, and synthesize information from different sources. 

C. Investigation of Problems concerns the use of Infonnation and Communication Technology 
to define and solve problems in core school subjects and in practical situations. 

D. Acknowledgment of Ideas and Information involves respect for the intellectual property of 
others and knowledge of how to credit others’ contributions appropriately, paying special 
attention to the misuse of information enabled by rapid technological advances. 

E. Selection and Use of Digital Tools includes both knowledge and skills for choosing 
appropriate tools and using a wide variety of electronic devices, including networked computing 
and communication technology and media. 

Resources Used in the Development of Assessment Targets 

The process of developing the assessment targets drew heavily on documents created over the 
past two decades by national experts in a wide variety of fields. Primary source documents 
include: 

• Benchmarks for Science Literacy ( AAAS, 1993); 

• Engineering in K-12 Education (NRC, 2009); 

• Framework for 21st Century Learning (Partnership for 2 1st Century Skills, 2007); 

• National Educational Technology Standards (ISTE, 2007); 

• National Science Education Standards (NRC, 1996); 

• Science for All Americans (AAAS, 1989); 

• Standards for Technological Literacy (ITEA, 2002); 
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• Standards for the 21st Century Learner (American Association of School Librarians, 

2007) . 

• Tech Tally: Approaches to Assessing Technological Literacy (NAE, 2006); and 

• Technically Speaking: Why All Americans Need to Learn More About Technology (NRC 
& NAE, 2002). 

Other documents that focused on science but recognized the importance of knowledge and skills 
in technology were valuable resources as well. These included: 

• The Assessment & Teaching of 2 1 st Century Skills, a Cisco/Intel/Microsoft project; 

• Benchmarking for Success: Ensuring U.S. Students Receive a World Class Education 
(National Governors Association, Council of Chief State School Officers, and Achieve, 

2008) ; 

• Best practices in various state frameworks on science, technology, and engineering; 

• Key Competencies for Lifelong Learning: European Reference Framework (European 
Communities, 2007); 

• PISA 2006: Science Competencies for Tomorrow ’s World (Organisation for Economic 
Co-operation and Development, 2007); 

• Science Framework for the 2009 National Assessment of Educational Progress (National 
Assessment Governing Board, 2008); and 

• Trends in International Mathematics and Science Study (TIMSS) (National Center for 
Education Statistics, 2008). 

The steering and planning committees recognize and appreciate the efforts of Achieve, Inc., the 
American Association for the Advancement of Science, the International Society for Technology 
in Education, the International Technology and Engineering Educators Association, the National 
Academy of Engineering, the National Research Council, and the Partnership for 21st Century 
Skills for their efforts in developing these source materials, for giving permission to quote the 
materials when desired, and for assisting in developing this framework. 

Technology and Society 


From the beginning of human culture, technology 
and society have been closely intertwined. From 
stone tools to computers and the Internet, 
technologies have allowed people to shape the 
physical world and the world of knowledge to meet 
their needs and wants, to extend the reach of their 
bodies, hands, and minds, to span rivers, and to 
traverse continents. From arrowheads to 
communication devices, technologies have always 
been an intrinsic part of civilization, and this is 
particularly true today, in the early part of the 2 1 st 
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century. This relationship is reflected in all of the national standards documents reviewed for this 
framework. It follows that awareness of the relationship between technology and society is an 
essential aspect of technology and engineering literacy. 

Essential knowledge and skills for this facet of technology and engineering literacy are divided 
into four subareas: 

A. Interaction of Technology and Humans; 

B. Effects of Technology on the Natural World; 

C. Effects of Technology on the World of Information and Knowledge; and 

D. Ethics, Equity, and Responsibility. 

A fundamental principle in the area of Interaction of Technology and Humans is that societies 
shape the technologies that are developed and used, and that those technologies in turn shape 
societies. Students are expected to demonstrate their understanding of the positive and negative 
effects that technologies may have on different aspects of society, as well as their capability to 
analyze historical and current examples of the technology-society relationship using concepts 
such as criteria, constraints, trade-offs, and consequences. Students should weigh societal and 
behavioral changes along with purely technological solutions. For example, encouraging the 
recycling and reuse of household materials may be more cost-effective than building new waste 
facilities. 

Effects of Technology on the Natural World takes a nuanced view of the relationship between 
technology and environmental change, recognizing both the negative impacts of technology on 
the environment and the ways in which people have used technology to restore and protect 
natural environments. Students are expected to recognize that technological decisions involve 
competing priorities and also to consider the consequences of alternative decisions in developing 
sustainable solutions to environmental problems. 

Effects of Technology on the World of Information and Knowledge addresses the increasing 
access permitted by information technology to expertise and information, the many powerful 
methods for storage and management of infonnation, the expansion of the capability to express 
ideas and representations of dynamic phenomena, and the support of distributed teamwork. 

The area of Ethics, Equity, and Responsibility addresses one of the most important aspects of 
technology and engineering literacy: the fact that technological decisions made by some people 
have significant impacts on others. Many of the thorniest technological issues in society concern 
effects that cross borders, such as acid rain, and many of them have global implications, such as 
the attribution and ownership of ideas and products and the effects of fossil fuel use on climate. 
The focus of this area is on general principles that can be applied when thinking about ethical 
issues that concern various technologies, although individual assessment items will inevitably 
tend to focus on a specific technology. The framework identifies the knowledge and skills that 
students should have for analyzing the issues, gathering evidence that could support multiple 
perspectives, and presenting alternative solutions to technological issues that have ethical 
implications. The framework does not take positions on controversial issues. 
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The following narrative provides an overview of each subarea, followed by tables that detail the 
knowledge and skills that will be assessed by the 2014 NAEP Technology and Engineering 
Literacy Assessment in the area of Technology and Society. 

A. Interaction of Technology and Humans 

Many students are first exposed to the interaction between technology and human society 
through the study of history. They leam about the “ages of civilization,” starting with the Stone 
Age, the Bronze Age, the Iron Age, the Industrial Age, and, most recently, the Information Age. 
So these students have already been provided with a number of examples of how societies meet 
their needs by transforming the natural materials in the world around them to create new 
technologies, and they have seen how these technologies in turn shape their societies and their 
relationship to other societies through such mechanisms as trade, communication, war, and 
assimilation. 

Students are also expected to know from history and from their personal experiences that the 
relationship between technology and society is reciprocal. Society drives technological change, 
while changing technologies in turn shape society. 

Although the effects of technological change are more difficult to discern when the period of 
observation is a few years rather than a number of centuries, students are still capable of 
reflecting on the technological changes that have occurred during their lifetimes. They should 
also be able to observe how the technological changes that are underway are driven by the needs 
of society, and they should be able to predict what some of the consequences of those new 
technologies might be. Examples of technological changes that nearly all students will have 
observed include new kinds of media, computers, and communication systems; the development 
of more fuel-efficient cars; the construction of new or improved buildings, roads, and bridges; 
and new foods and types of clothing. 

Key principles in the area of Interaction of Technology and Humans that all students can be 
expected to understand at increasing levels of sophistication are: 

• The relationship between technology and society is reciprocal. Society drives 
technological change, while changing technologies in turn shape society. 

• Technological decisions should take into account both costs and benefits. 

• When considering technological decisions that involve competing priorities, it is helpful 
to consider the trade-offs among alternative solutions. 

• Technologies may have unanticipated consequences, which become apparent only over 
time as the technology becomes more pervasive or powerful. 

• Technological solutions are developed and evaluated on the basis of criteria and 
constraints. 

Fourth graders are expected to know that people’s needs and desires detennine which 
technologies are developed or improved. For example, cellphones were invented, produced, and 
sold because people found it useful to be able to communicate with others wherever they were. 
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Students should also know that these new products, tools, and machines in turn affect the lives of 
individuals, families, and whole communities. An example is how transportation and 
communications systems enable people who live far apart to work together and interact with 
each other in new ways. 

Eighth graders are expected to understand how technologies and societies coevolve over 
significant periods of time. For example, the need to move goods and people across distances 
prompted the development of a long series of transportation systems from horses and wagons to 
cars and airplanes. They should also recognize that technologies may have effects that were not 
anticipated and that do not become apparent until the technology becomes widespread. For 
example, it was not until cellphones were widely used that it became apparent that people would 
use them while driving, creating an increased risk of accidents. Finally, eighth graders should be 
able to compare the effects of the same technology on different societies. 

Twelfth graders are expected to realize that the interplay between culture and technology is 
dynamic, with some changes happening slowly and others very rapidly. They should be able to 
use various principles of technology design — such as the concepts of trade-offs and unintended 
consequences — to analyze complex issues at the interface of technology and society, and to 
consider the implications of alternative solutions. For example, modem medical technologies 
have greatly increased average lifespan, but many of these new technologies are very costly and 
not available to everyone. The availability of advanced medical technologies also differs from 
country to country as a result of differing economic and political conditions and cultural values. 


2-7 



Technology and Engineering Literacy Framework for the 2014 NAEP 


Table 2.2 Interaction of Technology and Humans assessment targets for grades 4, 8, and 12 


A. Interaction of Technology and Humans 

Fourth graders should be aware of how products, tools, and machines affect communities and make it possible for 
people to work together. Eighth graders should understand how society drives technological change and how new or 
improved technologies affect a society’s economy, politics, and culture. Twelfth graders should have a heightened 
cultural sensitivity and attain a global view of the interplay between technology and culture. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

T.4.1: People’s needs and desires 
determine which new tools, 
products, and machines are 
developed and made available. 

Students know that: 

T.8.1: Economic, political, social, 
and cultural aspects of society drive 
improvements in technological 
products, processes, and systems. 

Students know that: 

T.12.1: The decision to develop a 
new technology is influenced by 
societal opinions and demands. 
These driving forces differ from 
culture to culture. 

T.4.2: The introduction of a new 
tool, product, or machine usually 
brings both benefits and costs, and it 
may change how people live and 
work. 

T.8.2: Technology interacts with 
society, sometimes bringing about 
changes in a society’s economy, 
politics, and culture, and often 
leading to the creation of new needs 
and wants. 

T.12.2: Changes caused by the 
introduction and use of a new 
technology can range from gradual 
to rapid and from subtle to obvious, 
and can change over time. These 
changes may vary from society to 
society as a result of differences in a 
society’s economy, politics, and 
culture. 

Students are able to: 

T.4.3: Identify potential positive and 
negative effects of the introduction 
of a new technology into a 
community. 

Students are able to: 

T.8.3: Describe and analyze positive 
and negative impacts on society 
from the introduction of a new or 
improved technology, including 
both expected and unanticipated 
effects. 

Students are able to: 

T.12.3: Choose an appropriate 
technology to help solve a given 
societal problem, and justify the 
selection based on an analysis of 
criteria and constraints, available 
resources, likely trade-offs, and 
relevant environmental and cultural 
concerns. 

T.4.4: Compare the effects of two 
different technologies on their own 
lives by imagining what their lives 
would be like without those 
technologies. 

T.8.4: Compare the impacts of a 
given technology on different 
societies, noting factors that may 
make a technology appropriate and 
sustainable in one society but not in 
another. 

T.12.4: Analyze cultural, social, 
economic, or political changes 
(separately or together) that may be 
triggered by the transfer of a specific 
technology from one society to 
another. Include both anticipated and 
unanticipated effects. 
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B. Effects of Technology on the Natural World 

As with technology’s influences on culture and society, the effects of a technology on the 
environment can be either positive or negative. Since the Industrial Revolution and the rapid 
growth of human populations, the potential for technology to have a major impact on the 
environment has grown. Consequently, an essential aspect of technology and engineering 
literacy is an understanding of certain key principles about the effects of technology on the 
natural environment and of the many important efforts that people have made to preserve natural 
habitats, reduce air and water pollution, and maintain a healthful environment. 

Individuals who are literate in technology and engineering should be aware of methods that have 
been developed to reduce the environmental impacts of technology. For example, an important 
step in designing a new product is to take the product’s life cycle into account. Such an analysis 
may start with the raw materials that need to be mined or grown, the industrial processes and 
energy needed to manufacture the product, the transportation technologies required to get it to 
market, and its eventual disposal when the product is no longer needed. 

Other ways to reduce environmental impact include the use of communication technologies to 
allow people to work at home rather than physically commute, the use of computer models to 
optimize industrial processes to conserve energy and reduce waste, and the expansion of 
alternative energy sources such as wind power. 

In finding a balance between technological development and environmental protection, a key 
overarching principle is that one should attempt, when possible, to find sustainable solutions. As 
defined by the Brundtland Commission in 1987, “sustainable solutions” are those that meet the 
needs of the present without compromising the ability of future generations to meet their own 
needs. 

Key principles in the area of Effects of Technology on the Natural World that all students 
can be expected to understand at increasing levels of sophistication are: 

• The use of technology may affect the environment positively or negatively. 

• Some technological decisions put environmental and economic concerns in competition 
with one another, while others have positive effects for both the economy and the 
environment. 

• Reusing, recycling, and using fewer resources can reduce environmental impacts. 

• Resources such as oceans, fresh water, and air, which are shared by everyone, need to be 
protected by careful planning and regulation of technological systems. 

• Some technologies can reduce the negative impacts of other technologies. 

• Sustainable solutions are those that meet the needs of the present without compromising 
the ability of future generations to meet their own needs (Brundtland Commission, 1987). 

At the fourth-grade level, students are expected to know that sometimes technology can cause 
environmental harm. For example, litter from food packages and plastic forks and spoons 
discarded on city streets can travel through stonn drains to rivers and oceans, where they can 
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harm or kill wildlife. However, such negative effects can be lessened by reusing or recycling 
products as well as by reducing the amount of resources used in producing the products. 

Eighth graders are expected to recognize that technology and engineering decisions often involve 
weighing competing priorities, and there are no perfect solutions. For example, dams built to 
control floods and produce electricity have left wilderness areas under water and affected the 
ability of certain fish to spawn. They should be able to analyze such conflicts and be able to 
recommend changes that would reduce environmental impacts. For example, students could 
study the trade-offs involved in using paper or plastic to carry groceries or research the causes 
and effects of acid rain on forests and the costs of reducing those effects. They should know that 
designers can reduce waste by taking the entire life cycle of a product into account during 
design. For example, students should be able to discuss what a community could do when its 
landfill is close to capacity or find ways that designers of new products could reduce waste by 
considering the life cycle of a product. 

By 12th, grade students should have had a variety of experiences in which technologies were 
used to reduce the environmental impacts of other technologies, such as the use of environmental 
monitoring equipment. Students should be able to analyze the effects of different technologies on 
the environment — for example, by using data on the environmental impacts of power plants that 
use different types of fuel to infonn decisions on which types of new power plants to build. 
Students should also be able to analyze complex human activities, such as energy generation, and 
propose sustainable solutions. Students could, for example, research the environmental impacts 
of energy generation and create a presentation to a United Nations council on the trade-offs of 
various solutions. 
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Table 2.3 Effects of Technology on the Natural World assessment targets for 

grades 4, 8, and 12 


B. Effects of Technology on the Natural World 

Fourth graders should be aware that the use of technology can affect the environment, that the environment affects 
technology, and that reusing and recycling products can avoid damaging the environment. Eighth graders should be 
able to investigate the environmental effects of alternative decisions by tracing the life cycle of products and 
considering the trade-offs involved in different technologies. Twelfth graders should be aware that technologies used 
to monitor environmental change can help inform decision-making, and they should also be able to investigate 
complex global issues and generate innovative, sustainable solutions. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

T.4.5: The use of technology can 
affect the environment, including 
land, water, air, plants, and animals. 
The environment also affects 
technology by providing sources of 
energy and raw materials. 

Students know that: 

T.8.5: Some technological decisions 
involve trade-offs between 
environmental and economic needs, 
while others have positive effects for 
both the economy and environment. 

Students know that: 

T.12.5: Many technologies have 
been designed to have a positive 
impact on the environment and to 
monitor environmental change over 
time to provide evidence for making 
informed decisions. 

T.4.6: Reusing and recycling 
materials can save money while 
preserving natural resources and 
avoiding damage to the 
environment. 

T.8.6: Resources such as oceans, 
fresh water, and air — which are 
essential for life and shared by 
everyone — are protected by 
regulating technologies in such areas 
as transportation, energy, and waste 
disposal. 

T.12.6: Development and 
modification of any technological 
system needs to take into account 
how the operation of the system will 
affect natural resources and 
ecosystems. 

Students are able to: 

T.4.7: Identify the impact of a 
specific technology on the 
environment and determine what 
can be done to reduce negative 
effects and increase positive effects. 

Students are able to: 

T.8.7: Compare the environmental 
effects of two alternative 
technologies devised to solve the 
same problem or accomplish the 
same goal and justify which choice 
is best, taking into account 
environmental impacts as well as 
other relevant factors. 

Students are able to: 

T.12.7: Identify a complex global 
environmental issue; develop a 
systematic plan of investigation; and 
propose an innovative, sustainable 
solution. 
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C. Effects of Technology on the World of Information and Knowledge 

Human civilization owes its current form to a number of major revolutions in the capacity to 
communicate and pass along information. The genesis of writing, a technological development 
that began at least 3,000 years ago in ancient Mesopotamia, led to a flowering of commerce, 
mathematics, science, and learning (Neugebauer, 1969; Van De Mieroop, 1999). Another 
milestone was the invention of the printing press by Johannes Gutenberg in the 15th century, 
which made it possible for ideas to be passed along to many people at widely distributed 
locations and times. Inventions of the telephone, telegraph, radio, movies, television, and the 
Internet all extended communication and expression. These revolutions changed the world of 
information and knowledge, resulting in transformative effects on society. 

Rapid advances in information and communication technologies during the latter half of the 20th 
and early 2 1 st centuries are creating an Information Age revolution. These technologies have 
made possible the storage, organization, and manipulation of vast quantities of data, far beyond 
what was possible for a physical library, and have greatly facilitated access to the information by 
anyone, anywhere. Together these technologies are modifying the world of information and 
knowledge itself, with implications for individuals, organizations, and entire societies. 

It is important for all citizens to understand the societal needs that led to the development of 
information and communication technologies and the effects, both positive and negative, of these 
technologies on the creation, extension, and use of knowledge and the expression of ideas. 
Students can acquire these insights and capabilities by taking courses in technology or simply 
through daily activities, such as studying the traditional subjects in school and pursuing personal 
interests at home. In other words, the process of using various technologies to access and interact 
with information and knowledge can yield valuable learning about the technology itself, and 
therefore contributes to students’ technology and engineering literacy. 

In contrast to the assessment targets later in this chapter under “Information and Communication 
Technology,” which are concerned with student knowledge and the use of various technologies, 
the assessment targets in this subarea emphasize the revolutionary consequences of the 
Information Age — the ways in which society affects what information and knowledge is 
available and how the availability of that information profoundly affects society. 
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Key principles in the area of Effects of Technology on the World of Information and 
Knowledge that all students can be expected to understand at increasing levels of sophistication 
are: 

• information technology is evolving rapidly, enabling ever-increasing amounts of 
information and data to be stored, managed, enhanced, analyzed, and accessed through a 
wide array of devices in various media formats. 

• Infonnation and Communication Technology (ICT) enables the creation and modification 
of information and knowledge products by remotely connected individuals and teams. 

• The emergence of intelligent infonnation technologies and the development of 
sophisticated modeling and simulation capabilities are transfonning the world of 
information and knowledge, with potentially profound effects on society. 

Fourth-grade students should know that information technology provides access to vast amounts 
of information, that it can also be used to modify and display data, and that communication 
technologies make it possible to communicate across great distances using writing, voice, and 
images. They should be able to identify examples of positive and negative impacts of these tools. 
For example, students should be able to identify positive effects of being able to send data to 
others. They should also be able to identify negative effects such as hasty email responses. 

Eighth graders should be aware of the rapid progress in development of ICT, should know how 
information technologies can be used to analyze, display, and communicate data, and should be 
able to collaborate with other students to develop and modify a knowledge product. For example, 
students should understand that translation tools on personal communication devices permit 
collaboration with students from other countries on a school project, such as digital storytelling. 
Eighth graders should also understand that problems can be caused by using digital information 
or video without verifying quality. 

By 12th grade, students should have a full grasp of the types of data, expertise, and knowledge 
available online and should be aware of intelligent information technologies and the uses of 
simulation and modeling. They should also understand the potential disadvantage of uncritical 
use of the technologies. For example, they should know that false infonnation can spread rapidly 
through the Internet and that these falsehoods, once established, can be difficult to correct. 
Students should be aware that the ubiquitous use of information communication and 
dissemination affects governments, news, and other organizations as well as individuals, and 
they should understand the extent to which ICT has enabled a revolution in the world of 
knowledge. 
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Table 2.4 Effects of Technology on the World of Information and Knowledge 
assessment targets for grades 4, 8, and 12 


C. Effects of Technology on the World of Information and Knowledge 

Fourth graders should know that information, knowledge, and expertise can be accessed, collected, and shared by 
using information and communication technologies with positive and negative effects. Eighth-grade students 
should be able to critically weigh how information and communication technologies are rapidly evolving and 
changing the ways that people interact with information and each other. Twelfth graders should know that 
sophisticated information and communication technologies have transformed the world of information and 
knowledge itself, with positive and negative implications for society. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

T.4.8: Information technology 
provides access to vast stores of 
knowledge and information. This 
can result in positive and negative 
effects. 

Students know that: 

T.8.8: Information technologies are 
developing rapidly so that the 
amount of data that can be stored 
and made widely accessible is 
growing at a faster rate each year. 

Students know that: 

T.12.8: Information technology 
allows access to vast quantities of 
data, expertise, and knowledge 
through a wide array of devices and 
formats to answer questions, solve 
problems, and inform the decision- 
making process. 

T.4.9: Information technologies can 
be used to modify and display data 
in various ways that can be helpful 
or deceptive. 

T.8.9: Information technologies 
make it possible to analyze and 
interpret data — including text, 
images, and sound — in ways that are 
not possible with human senses 
alone. These uses may result in 
positive or negative impacts. 

T.12.9: Information technologies 
such as artificial intelligence, image 
enhancement and analysis, and 
sophisticated computer modeling 
and simulation create new types of 
information that may have profound 
effects on society. These new types 
of information must be evaluated 
carefully. 

T.4.10: Communications 
technologies make it possible for 
people to communicate across large 
distances in writing, voice, and 
images. 

T.8.10: The large range of personal 
and professional information 
technologies and communication 
devices allows for remote 
collaboration and rapid sharing of 
ideas unrestricted by geographic 
location. 

T.12.10: The development of 
communication technologies that 
enable people to access vast 
quantities of information and publish 
their ideas globally has implications 
for governments, organizations, and 
individuals. 

Students are able to: 

T.4.11: Use information and 
communications technologies to 
access and interpret data and 
communicate with others. 

Students are able to: 

T.8.11: Use appropriate information 
and communication technologies to 
collaborate with others on the 
creation and modification of a 
knowledge product that can be 
accessed and used by other people. 

Students are able to: 

T.12.11: Give examples to illustrate 
the effects on society of the 
recording, distribution, and access to 
information and knowledge that 
have occurred in history, and discuss 
the effects of those revolutions on 
societal change. 
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D. Ethics, Equity, and Responsibility 

Although technological advances have improved quality of life, newer technologies have 
sometimes resulted in negative effects, which in turn may have various ethical implications. 
Consequently, it is becoming increasingly important for citizens to recognize ethical issues 
related to the introduction and use of various technologies. For example, factories and power 
plants that benefit the citizens of one country may produce gases that cause acid rain, damaging 
forests in that country and neighboring countries. An ethical response to such a situation starts 
with the recognition that such effects are occurring, followed by concrete steps to mitigate the 
problem. 

The term “ethics” generally refers to a code of behavior or set of rules or guidelines for 
distinguishing between right and wrong. In general, there is no such thing as a universally 
agreed-on set of ethical guidelines, since a behavior considered ethical in one culture or 
professional group may be considered unethical by a different group of people. Nonetheless, 
there are some ways of behaving and thinking that most people consider ethical, such as honesty 
and integrity (Resnik, 2007). The tenn “equity” generally refers to fairness, or equality of 
opportunity, while “responsibility” generally means holding oneself accountable to accomplish 
certain things, or to be trustworthy. 

While ethics, equity, and responsibility are important considerations in all human endeavors, 
they are especially important for the design, production, inspection, and use of technologies, as 
these have an immediate effect on people’s lives. Innumerable examples come to mind, such as 
the collapse of buildings during earthquakes, the failure of bridges due to poor maintenance, the 
distribution of poisonous foods due to unsafe processing methods, and lapses in the testing and 
correction of mechanical failures in automobiles. Prevention of such human tragedies involves 
discussion of ethical practices and the responsibilities of individuals from technicians to 
engineers and policymakers. 

One sector of the current technological infrastructure that is especially vulnerable to unethical 
behaviors is the telecommunications sector and, in particular, the Internet. Access to the Internet 
offers unprecedented opportunities as well as challenges for students, as they have opportunities 
not only to access information but also to contribute and publish their own information for 
anyone in the world to read. But to use these tools (and others yet to be developed) in a 
responsible manner, students need to understand fundamental rules of ethical behavior with 
regard to the exchange of information. They also need to know how to protect themselves and to 
take personal responsibility for doing so. 

Key principles in the area of Ethics, Equity, and Responsibility that all students can be 
expected to understand at increasing levels of sophistication are: 

• Technology by itself is neither good nor bad, but its use may affect others. 

• Not everyone has access to the same technologies. 

• Differences in available technologies within the United States and in other countries have 
consequences for public health and prosperity. 
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• People living in one area need to be aware of how their use of technology affects the lives 
of people in other areas. 

• Storing infonnation digitally requires a heightened attention to remote security threats. 

• It is important for people to take responsibility for the appropriate use of technology. 

Fourth graders should recognize that tools and machines can be helpful or harmful. For example, 
cars are very helpful for going from one place to another quickly, but their use can lead to 
accidents in which people are seriously injured. Students should also recognize that technology 
can be used in ways that hurt others, such as when a false rumor is posted about someone online. 

Eighth graders should be able to recognize that the potential for misusing technologies always 
exists and that the possible consequences of such misuse must be taken into account when 
making decisions. They should have a grasp of the technological inequalities around the world — 
as illustrated by the existence of countries where few people can afford refrigerators and few 
schools have computers — and students should understand the economic and cultural reasons for 
these inequalities. They should know how to reduce the negative impacts that their use of 
technology may have on people who live in other areas. For example, they might consider 
avoiding the use of products that have been produced in ecologically vulnerable areas, such as 
the Amazonian rainforest. They should also have a solid understanding of a range of unethical 
and criminal behaviors involving the use of Internet and communications technologies. 

Twelfth graders should be able to take into account intended and unintended consequences in 
making technological decisions. They should understand the worldwide inequalities in 
technology access and know that efforts to transplant a technology from one culture to another 
should not be undertaken without a consideration of the costs and benefits to the society 
receiving the technology. They should be able to analyze the ethical responsibilities of various 
people in government and commercial enterprises, and demonstrate prudent and ethical use of 
communications technologies. 
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Table 2.5 Ethics, Equity, and Responsibility assessment targets for grades 4, 8, and 12 


D. Ethics, Equity, and Responsibility 

Fourth graders should know that tools and machines used carelessly might harm others, take responsibility for the 
appropriate use of tools and machines, and recognize misuses of communications and other technologies. Eighth 
graders should recognize that the same technologies are not available to everyone, and should take responsibility to 
reduce the negative impacts of technologies and increase their positive impacts. Twelfth graders should be able to 
take into account different viewpoints, recognize that transferring technologies from one society to another can be 
complex, and consider the consequences of unethical uses of technology. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

T.4.12: When using tools and 
machines, the results can be helpful 
or harmful. 

Students know that: 

T.8.12: Technology by itself is 
neither good nor bad, but its use 
may affect others; therefore, 
decisions about products, processes, 
and systems must take possible 
consequences into account. 

Students know that: 

T.12.12: Decisions made about the 
use of a technology may have 
intended and unintended 
consequences, and these 
consequences may be different for 
different groups of people and may 
change over time. Decisions about 
the use of a technology should 
consider different points of view. 

T.4.13: The technologies that people 
have available for essential tasks 
such as farming, cooking, medicine, 
transportation, and communication 
are vastly different in different parts 
of the world. 

T.8.13: People who live in different 
parts of the world have different 
technological choices and 
opportunities because of such 
factors as differences in economic 
resources, location, and cultural 
values. 

T.12.13: Disparities in the 
technologies available to different 
groups of people have consequences 
for public health and prosperity, but 
deciding whether to introduce a new 
technology should consider local 
resources and the role of culture in 
acceptance of the new technology. 

Students are able to: 

T.4.14: Explain the benefits and 
safe use of a tool or machine by 
showing how it can and should be 
used as well as how it should not be 
used and the consequences that may 
result if it is used inappropriately. 

Students are able to: 

T.8.14: Explain that it is important 
for citizens to reduce the negative 
impacts and increase the positive 
impacts of their technologies on 
people in another area or on future 
generations. 

Students are able to: 

T.12.14: Analyze responsibilities of 
individuals and groups, ranging from 
citizens and entrepreneurs to 
political and government officials, 
with respect to a controversial 
technological issue. 

T.4.15: Demonstrate the ethical use 
of information technologies by 
recognizing the ways that someone 
might harm someone else through 
the misuse of communication 
technologies, and the kinds of 
information that could lead to abuse 
if widely shared. 

T.8.15: Explain why it is unethical 
to infect or damage other people’s 
computers with viruses or “hack” 
into other computer systems to 
gather or change information. 

T.12.15: Demonstrate the 
responsible and ethical use of 
information and communication 
technologies by distinguishing 
between kinds of information that 
should and should not be publicly 
shared and describing the 
consequences of a poor decision. 
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Design and Systems 

We live in a global society that is increasingly 
dependent on technology. In the drive to satisfy 
human needs and wants, people have developed and 
improved ways to communicate, move people and 
goods, build structures, make products, enhance 
ideas, cure diseases, use energy, and provide 
nutritious and safe food and water, as well as 
numerous other innovations. Technological 
development has resulted in a vast network of 
products and systems — often called “the designed 
world” — that is constantly changing. The study of 
engineering design and systems is the study of the 
world in which all humans live and which all 
students will shape by the decisions they make as 
workers, consumers, and citizens. 

Because students live in a complex technological world, they face decisions every day that 
involve technology. Some of these are simple choices, such as deciding whether to use paper, 
plastic, or reusable bags for groceries or choosing which form of entertainment to enjoy, and 
others are more far-reaching and complex, such as which type of job to choose or what sort of 
medical treatment to select. How well students are prepared to make those choices depends in 
part on their understanding of technology. Essential knowledge and skills in this area of 
technology and engineering literacy are divided into four subareas: 

A. Nature of Technology; 

B. Engineering Design; 

C. Systems Thinking; and 

D. Maintenance and Troubleshooting. 

Understanding the Nature of Technology requires that one take a broad view. Simply put, 
technology satisfies the basic human needs for food and water, protection from the elements, 
health, energy, improved transportation, better and cheaper products, and improved 
communication. Students are expected to understand that the laws of nature provide limits on the 
types of technologies that can be developed. No one can create a perpetual motion machine, for 
example, as machines always require more energy input than they provide as useful output. 

Students are also expected to distinguish between science, technology, and engineering, and to 
recognize that science enables improvements in technology, while technological improvements 
created by engineers often lead to advances in science. Students should also recognize that some 
problems can be solved through behavioral rather than physical changes, for example, by 
encouraging the use of carpools to relieve traffic congestion rather than constructing additional 
highway lanes. 
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Engineering Design is an iterative and systematic approach to creating solutions to a broad 
variety of problems to meet people’s needs and desires. The process of design includes defining 
problems in terms of criteria and constraints; researching and generating ideas; selecting between 
or among alternatives; making drawings, models, and prototypes; optimizing, testing, evaluating 
the design, and redesigning if needed; and, eventually, communicating the results. 

Systems Thinking concerns the capability to identify the components, goals, and processes of 
systems. It also entails an understanding of such systems principles as feedback and control and 
also the ability to use simulations or other tools to predict the behavior of systems. 

Maintenance and Troubleshooting are how most people encounter technology on a daily basis — 
by troubleshooting technologies that malfunction and by maintaining tools and systems so they 
do not break down. The better a person understands the way something works, the easier it is to 
maintain it and to track down problems when they arise. 

Each of the above subareas relates to one of the broad categories included in the ITEEA 
Standards for Technological Literacy. A table illustrating these connections is presented in 
appendix F. 

A. Nature of Technology 

Two of every three people in the United States think “technology” means computers and the 
Internet (Rose & Dugger, 2002; Rose, Gallup, Dugger, & Starkweather, 2004). Some people 
conceptualize technology somewhat more broadly to include cellphones and other electronics. 
However, technology includes every way in which people manipulate the natural environment to 
satisfy their needs and wants. Frozen foods, paper cups, and clothing are examples of 
technology, as are dams, motorcycles, windmills, water treatment plants, flu shots, and 
grandfather clocks. Technology includes all of the various devices and systems that people make 
to fulfill some function. 

In addition to understanding the scope of technology, students are expected to understand how 
technology evolved and why the pace of technological change is so much faster today than in the 
past. For much of human history, technological knowledge was held by small groups of 
individuals who did not share it, but rather passed it guardedly from one generation to the next, 
sometimes from parent to child or master to apprentice. Today, by contrast, know-how is 
disseminated much more freely through a wide variety of educational institutions, both 
physically and online. Engineers and designers improve existing technologies, invent new 
devices and systems, and make technological breakthroughs that can be widely communicated in 
a short period of time, resulting in changes that can revolutionize entire industries. This is part of 
the reason that the rate of technological development is increasing at an unprecedented speed. 

Another part of the reason can be found in today’s rapid advances in science. In many cutting- 
edge fields, such as bioengineering and nanotechnology, scientists and engineers work hand in 
hand, and sometimes the roles of scientist and engineer are taken on by a single person. An 
example of science pushing technology can be found in the breakthroughs in genetics that have 
made possible new crops with higher yields and greater resistance to disease. Examples of 
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technology pushing science can be found in the way that engineers provide more precise 
instruments, better collaboration tools, and evermore powerful computers. 

Tools and materials have also improved over time. From hand tools and power tools to computer 
probes and simulations, tools extend human capabilities, allowing people to see further and in 
greater detail, accomplish tasks more efficiently, and accomplish things that might otherwise be 
impossible. At the same time, new ways are constantly being developed to process raw materials 
to create products with properties unlike any in nature — self-cleaning clothing and paint, nano- 
fiber clothing that sheds water and never wrinkles, and composite materials for airplanes that are 
lighter and stronger than metal alloys, to name only a few. 

Key principles in the area of Nature of Technology that all students can be expected to 
understand at increasing levels of sophistication are: 

• Technology is constrained by laws of nature, such as gravity. 

• Scientists are concerned with what exists in nature; engineers modify natural materials to 
meet human needs and wants. 

• Technological development involves creative thinking. 

• Technologies developed for one purpose are sometimes adapted to serve other purposes. 

• Science, technology, engineering, mathematics, and other disciplines are mutually 
supportive. 

• The pace of technological change has been increasing. 

• Tools help people do things efficiently, accurately, and safely. 

Fourth-grade students are expected to distinguish natural and human-made materials, be familiar 
with simple tools, and recognize the vast array of technologies around them. 

Eighth graders should know how technologies are created through invention and innovation, 
should recognize that sometimes a technology developed for one purpose is later adapted to other 
purposes, and should understand that technologies are constrained by natural laws. They should 
also know that other resources besides tools and materials — energy, people, capital, and time — 
are generally needed to solve problems and meet design challenges. 

Twelfth graders should have an in-depth understanding of the ways in which technology 
coevolves with science, mathematics, and other fields; should be able to apply the concept of 
trade-offs to resolve competing values; and should be able to identify the most important 
resources needed to carry out a task. 
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Table 2.6 Nature of Technology assessment targets for grades 4, 8, and 12 


A. Nature of Technology 

Fourth graders should know that technology involves tools, materials, and creative thinking used to meet human 
needs and wants. Eighth graders should know that technology advances through invention and innovation and 
requires a variety of resources. Twelfth graders should know how technology coevolves with science and other 
fields to allow people to accomplish challenging tasks. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

D.4.1: Scientists ask questions 
about the world; engineers 
create and modify technologies 
to meet people’s needs and 
desires. 

Students know that: 

D.8.1: Science is the systematic 
investigation of the natural world. 
Technology is any modification of the 
environment to satisfy people’s needs 
and wants. Engineering is the process 
of creating or modifying technologies 
and is constrained by physical laws and 
cultural norms, and economic 
resources. 

Students know that: 

D.12.1: Advances in science have 
been applied by engineers to design 
new products, processes, and 
systems, while improvements in 
technology have enabled 
breakthroughs in scientific 
knowledge. 

D.4.2: The improvement of 
existing technologies and the 
development of new 
technologies involve creative 
thinking. 

D.8.2: Technology advances through 
the processes of innovation and 
invention. Sometimes a technology 
developed for one purpose is adapted to 
serve other purposes. 

D.12.2: Engineers use science, 
mathematics, and other disciplines 
to improve technology, while 
scientists use tools devised by 
engineers to advance knowledge in 
their disciplines. This interaction 
has deepened over the past century. 

D.4.3: Tools are simple objects 
that help people do things better 
or more easily, such as the 
cutting, shaping, and combining 
of materials that occur when 
making clothing. 

D.8.3: Tools have been improved over 
time to do more difficult tasks and to 
do simple tasks more efficiently, 
accurately, or safely. Tools further the 
reach of hands, voices, memory, and 
the five human senses. 

D.12.3: The evolution of tools, 
materials, and processes has played 
an essential role in the development 
and advancement of civilization, 
from the establishment of cities and 
industrial societies to today’s global 
trade and commerce networks. 

Students are able to: 

D.4.4: Inspect materials with 
different properties and 
determine which is most suitable 
for a given application. 

Students are able to: 

D.8.4: Simulate tests of various 
materials to determine which would be 
best to use for a given application. 

Students are able to: 

D.12.4: Take into account trade- 
offs among several factors when 
selecting a material for a given 
application. 

D.4.5: Choose an appropriate 
tool for accomplishing a task. 

D.8.5: Redesign an existing tool to 
make it easier to accomplish a task. 

D.12.5: Design a new tool to 
accomplish a task more efficiently. 
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B. Engineering Design 

Engineering design (sometimes called technological design) is an iterative, systematic process 
for solving problems that involves creativity, experience, and accumulated disciplinary 
knowledge. As used in this framework, engineering design is a broad term, including processes 
such as architectural design, manufacturing design, industrial design, and software design. 

Much like scientific inquiry, engineering design is a dynamic process, not a rigid method. As 
engineering and science are often confused, it is helpful to draw a distinction. Scientific inquiry 
begins with a question and proceeds to generate and test hypotheses until the question is 
answered. In contrast, engineering design begins with a problem and proceeds to generate and 
test solutions until a preferred solution or solutions are reached. Whereas science seeks to 
understand, engineering seeks to meet people’s needs. 

The engineering design process usually begins by stating a need or want as a clearly defined 
challenge in the form of a statement with criteria and constraints. For example, a group of 
engineers might be given the task of designing a cellphone with a particular set of features, of a 
particular size and weight, with a certain minimum battery life, and that is able to be 
manufactured at a particular cost. Criteria are characteristics of a successful solution, such as the 
desired function or a particular level of efficiency. Constraints are limitations on the design, such 
as available funds, resources, or time. Together, the criteria and constraints are referred to as the 
requirements for a successful solution. 

Once the challenge is defined, the next steps are often to investigate relevant scientific and 
technical information and the way that similar challenges have been solved in the past, and then 
to generate various possible solutions. This generation of potential solutions is the most creative 
part of the design process and is often aided by sketching and discussion. Using a process of 
informed decision-making, the designer or design team compares different solutions to the 
requirements of the problem and either chooses the most promising solution or synthesizes 
several ideas into an even more promising potential solution. The next step is usually to try out 
the solution by constructing a model, prototype (first of its kind), or simulation and then testing it 
to see how well it meets the criteria and falls within the constraints. An additional characteristic 
of engineering design is that ideas are tested before investing too much time, money, or effort. 

A person does not have to be an engineer to employ an engineering design process. Children can 
use this process to create a new toy, teachers can use it to plan a semester of lessons, and anyone 
can use it to address a need or desire encountered in everyday life. 

The result of an engineering design process is not always a product. In some cases the result may 
be a process (such as a chemical process for producing an improved paint), a system (such as an 
airline control system or a railway schedule), or a computer program (such as a video game or 
software to forecast the weather or model financial markets). 

When designing, it is important to take into account the entire life cycle of the product or 
process, including maintenance, troubleshooting, potential failure modes, impacts on the 
environment, and effects on society. Designing usually concludes with a presentation to clients 
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or other interested parties (often classmates) on the preferred solution. 

Optimization, which is sometimes part of designing, means finding the best possible solution 
when some criterion or constraint is identified as the most important and others are given less 
weight. For example, optimizing the design of a pen might mean designing for lowest cost, best 
ink flow, or best grip, but not all three. Optimizing the design of an airplane engine usually 
means maximizing safety. In some engineering disciplines, the entire engineering process is 
referred to as “optimization under constraint.” 

It bears emphasizing that engineering design is not a rigid method. Different instructional 
materials define it differently, although most definitions specify a sequence of steps, ranging 
from 5 steps for elementary students to 8 or 10 steps for high school students. The steps need not 
be followed in order. An experienced engineer might skip ahead a step or two or go back one or 
two steps. Or after generating solutions, it may become clear that the problem was poorly 
defined, and so it is best to restart the process from the beginning. Regardless of these 
differences in definition and use, engineering design always begins with a problem and ends with 
one or more solutions. 

Key principles in the area of Engineering Design that all students can be expected to 
understand at increasing levels of sophistication are: 

• Engineering design is a systematic, creative, and iterative process for addressing 
challenges. 

• Designing includes identifying and stating the problem, need, or desire; generating ideas; 
evaluating ideas; selecting a solution; making and testing models or prototypes; 
redesigning; and communicating results. 

• Requirements for a design challenge include the criteria for success, or goals to be 
achieved, and the constraints or limits that cannot be violated in a solution. Types of 
criteria and constraints include materials, cost, safety, reliability, performance, 
maintenance, ease of use, aesthetic considerations, and policies. 

• There are several possible ways of addressing a design challenge. 

• Evaluation means determining how well a solution meets requirements. 

• Optimization involves finding the best possible solution when some criterion or 
constraint is identified as the most important and other constraints are minimized. 

• Engineering design usually requires one to develop and manipulate representations and 
models (for example, prototypes, drawings, charts, and graphs). 

Fourth graders should know that engineering design is a purposeful method of solving problems 
and achieving results. They should be able to state a simple design challenge in their own words, 
test a solution, and communicate the findings with drawings and models. 

Eighth graders should be able to carry out a full engineering design process to solve a problem of 
moderate difficulty. They should be able to define the challenge in tenns of criteria and 
constraints, research the problem, generate alternative solutions, build and test a model or 
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prototype, redesign, and communicate the findings. 

Twelfth graders should be able to meet a complex challenge, weigh alternative solutions, and use 
the concept of trade-offs to balance competing values. They should be able to redesign so as to 
arrive at an optimal solution. 
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Table 2.7 Engineering Design assessment targets for grades 4, 8, and 12 


B. Engineering Design 

Fourth graders should start to answer the question “How are technologies created?” by confronting simple yet 
systematic design challenges. Eighth graders should be able to use a more elaborate engineering design process, 
including problem definition, the use of prototypes, testing and iteration, and trade-offs. Twelfth graders should have 
a deep understanding and a broad array of design skills, including optimization. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

D.4.6: Engineering design is a 
systematic and creative process for 
meeting challenges. Often there 
are several solutions to a design 
challenge. Each one might be 
better in some way than the others. 
For example, one solution might 
be safer, while another might cost 
less. 

Students know that: 

D.8.6: Engineering design is a 
systematic, creative, and iterative 
process for meeting human needs and 
wants. It includes stating the 
challenge, generating ideas, choosing 
the best solution, making and testing 
models and prototypes, and 
redesigning. Often there are several 
possible solutions. 

Students know that: 

D.12.6: Engineering design is a 
complicated process in which 
creative steps are embedded in 
content knowledge and research on 
the challenge. Decisions on trade- 
offs involve systematic comparisons 
of all costs and benefits, and final 
steps may involve redesigning for 
optimization. 

D.4.7: Requirements for a design 
include the desired features of a 
product or system as well as the 
limits placed on the design, such 
as which materials are available. 

D.8.7: Requirements for a design are 
made up of the criteria for success and 
the constraints, or limits, which may 
include time, money, and materials. 
Designing often involves making 
trade-offs between competing 
requirements and desired design 
features. 

D.12.7: Specifications involve 
criteria, which may be weighted in 
various ways, and constraints, which 
can include natural laws and 
available technologies. Evaluation is 
a process for determining how well a 
solution meets the requirements. 

Students are able to: 

D.4.8: Use a systematic process to 
design a solution to a simple 
problem. 

Students are able to: 

D.8.8: Carry out a design process to 
solve a moderately difficult problem 
by identifying criteria and constraints, 
determining how they will affect the 
solution, researching and generating 
ideas, and using trade-offs to choose 
between alternative solutions. 

Students are able to: 

D.12.8: Meet a sophisticated design 
challenge by identifying criteria and 
constraints, predicting how these will 
affect the solution, researching and 
generating ideas, and using trade-offs 
to balance competing values in 
selecting the best solution. 

D.4.9: Construct and test a simple 
model to determine if it meets the 
requirements of a problem. 

D.8.9: Construct and test a model and 
gather data to see if it meets the 
requirements of a problem. 

D.12.9: Construct and test several 
models to see if they meet the 
requirements of a problem. Combine 
features to achieve the best solution. 

D.4.10: Communicate design 
ideas using drawings and models. 

D.8.10: Communicate the results of a 
design process and articulate the 
reasoning behind design decisions by 
using verbal and visual means. 
Identify the benefits of a design as 
well as the possible unintended 
consequences. 

D.12.10: Communicate the entire 
design process from problem 
definition to evaluation of the final 
design, taking into account relevant 
criteria and constraints, including 
aesthetic and ethical considerations 
as well as purely logical decisions. 
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C. Systems Thinking 

A system is any collection of interacting parts that make up a whole. In a sense, all technologies 
can be thought of as systems. Furthermore, the ways in which objects are produced and used can 
also be thought of as systems, as technological objects all have a life cycle, being made from raw 
materials, used, and eventually being discarded, at which point they may be recycled or added to 
landfills. More broadly, there are many examples of systems that are not technological in origin 
or else are only partly technology: ecosystems, financial systems, political systems, and so on. 

Beyond understanding that systems exist, citizens who are literate in technology and engineering 
should be comfortable with the broader skill of systems thinking — a set of cognitive tools that 
increases in sophistication and power over time. Systems thinking is the capability to 
investigate — or think about — a system using certain principles. It enables people to understand 
complicated situations that involve many interactions. For example, consider these principles: 
systems include subsystems; any given system is typically part of one or more larger systems; 
and systems interact with other systems. 

These principles are important in thinking about the nation’s transportation system, among other 
things. The transportation system consists of a vast network of roads and rails and millions of 
vehicles. It is dependent on a second system that extracts oil from wells halfway around the 
world and carries that oil in thousands of supertankers to huge refineries, and from there to 
distribution points and gas stations. The combustion of fuels produces carbon dioxide; these 
systems affect the global climate system as well. Citizens who understand the effects of different 
fuels on the environment will be able to make decisions about what kind of automobile to 
purchase based on both these interconnected technological systems and the price of gas. 

Systems thinking can be applied equally well to understanding systems other than purely 
technological ones — for example, to analyze the way that communication technologies influence 
society and in turn are influenced by society, to think about the interplay between energy 
technologies and global climate change, and to predict future conditions given current trends and 
changes people may choose to make. It is a cognitive tool that helps people analyze problems 
they encounter in various settings and to propose solutions or detennine reasonable courses of 
action. Simply put, systems thinking helps people understand how things are put together, how 
they function, and how they connect with other parts of the world, and it assists people in making 
informed decisions. 

Key principles in the area of Systems Thinking that all students can be expected to understand 
at increasing levels of sophistication are: 

• Technological systems have parts that work together to accomplish a goal. 

• Systems may include subsystems and may interact with other systems. Systems may also 
be embedded within larger systems. 

• Dynamic technological systems require energy with more complicated systems tending to 
require more energy and to be more vulnerable to error and failure. 
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• Technological systems are designed for specific purposes. They incorporate various 
processes that transform inputs into outputs. Two important features of technological 
systems are feedback and control. 

• Various methods can be used to increase the reliability of technological systems. 

Fourth graders should know that a system is a collection of interacting parts that make up a 
whole, that systems require energy, and that systems can be either living or nonliving. They 
should also be able to look at a simple system, identify its various parts, and recognize the 
functions of these parts within the larger system. 

Eighth graders should be able to analyze a technological system in terms of goals, inputs, 
processes, outputs, feedback, and control. They should be aware that systems can interact with 
each other and be able to identify the subsystems and components of a device by using reverse 
engineering — the process of analyzing a system to see how it works in order to design a different 
device that performs the same function. Eighth graders should also be able to trace the life cycle 
of a product from raw materials to eventual disposal. 

Twelfth graders should be aware that technological systems are the product of goal-directed 
designs and that the building blocks of any technology consist of systems that are embedded 
within larger technological, social, and environmental systems. They should also be aware that 
the stability of a system is influenced by all of its components, especially those in a feedback 
loop. (Negative feedback tends to stabilize systems while positive feedback leads to instability.) 
Students should be able to use various techniques to forecast what will happen if a component or 
process is changed. 
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Table 2.8 Systems Thinking assessment targets for grades 4, 8, and 12 


C. Systems Thinking 

Fourth graders should be able to identify systems, subsystems, components, and boundaries in their everyday world 
and construct simple systems designed to accomplish particular goals. Eighth graders should be able to describe 
goals, inputs, outputs, and processes of systems; use reverse engineering and life cycles to analyze systems in terms 
of feedback and the flow of energy; and modify and construct moderately complicated systems. Twelfth graders 
should understand that systems are embedded in larger systems, recognize factors that stabilize systems, use systems 
for forecasting, and redesign complicated systems to improve reliability. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

D.4.11: All technological systems 
require energy and have parts that 
work together to accomplish a goal. 

Students know that: 

D.8.11: Technological systems are 
designed to achieve goals. They 
incorporate various processes that 
transform inputs into outputs. They 
all use energy in some form. These 
processes may include feedback and 
control. 

Students know that: 

D.12.11: The stability of a system 
depends on all of its components and 
how they are connected, with more 
complicated systems tending to 
require more energy and to be more 
vulnerable to error and failure. 
Negative feedback loops tend to 
increase the stability and efficiency 
of systems. 

D.4.12: Many systems have 
subsystems within them and are 
defined by boundaries. Many 
systems are parts of larger systems. 

D.8.12: Technological systems can 
interact with one another to perform 
more complicated functions and 
tasks than any individual system can 
do by itself. 

D.12.12: Technological systems are 
embedded within larger 
technological, social, natural, and 
environmental systems. 

Students are able to: 

D.4.13: Given a product, identify its 
systems, subsystems, and 
components by taking it apart. 

Students are able to: 

D.8.13: Examine a product or 
process through reverse engineering 
by taking it apart step by step to 
identify its systems, subsystems, and 
components, describing their 
interactions, and tracing the flow of 
energy through the system. 

Students are able to: 

D.12.13: Examine a system to 
predict how it will perform with a 
given set of inputs in a given 
situation and how performance will 
change if the components or 
interactions of the system are 
changed. 

D.4.14: Create a diagram of a 
machine that contains multiple 
subsystems. Label the subsystems to 
explain what each one does. 

D.8.14: Measure and compare the 
production efficiency of two 
machines, a simple machine and a 
complex machine, designed to 
accomplish the same goal. 

D.12.14: Redesign a complex 
machine by modifying or 
rearranging its subsystems in order 
to optimize its efficiency. 

D.4.15: Construct a simple system 
to accomplish a goal, based on 
knowledge of the function of 
individual components. 

D.8.15: Construct and use a 
moderately complicated system, 
given a goal for the system and a 
collection of parts, including those 
that may or may not be useful in the 
system. 

D.12.15: Construct and test a 
manufacturing system composed of 
several machines to accomplish a 
given goal. Redesign the system to 
optimize its efficiency. 
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D. Maintenance and Troubleshooting 

The statement that “anything that can go wrong will go wrong,” known as Murphy’s Law, has 
been attributed to aerospace engineer Edward Murphy, who first used the expression (or 
something like it) in 1949 to explain the failure of measurement equipment for a high-speed 
rocket sled (Spark, 2006). Murphy’s Law has come to characterize everyday life, not only for 
engineers but also for everyone in modern society. Today humans are surrounded by and 
dependent on complicated devices that seem to go wrong at critical times. It is not uncommon to 
experience more than one technological failure in a single day, whether it’s a car that fails to 
start, a cellphone without “bars,” or something as simple as an eyeglass frame with a lens that 
repeatedly pops out. 

A person who is literate in technology and engineering is aware that all technological systems 
fail at one time or another and is therefore equipped with a foundation of concepts and skills that 
can be applied either to correct failed devices and systems or to prevent the failure from 
occurring in the first place. The most important of these concepts and skills are maintenance and 
troubleshooting. 

In the 2014 NAEP Technology and Engineering Literacy Framework, the term “maintenance” 
has a very specific meaning: It refers to keeping technological devices and systems in good 
condition so as to extend their useful life and reduce the number of breakdowns. For example, 
maintenance can refer to the regular upkeep of technologies so that they are less likely to fail, 
such as periodically replacing the oil in a car engine, cleaning the lint filter of a clothes dryer, or 
running regular software updates on a computer operating system. 

Troubleshooting, by contrast, refers to a systematic method of dealing with failures once they 
have occurred. It is common to begin troubleshooting by ascertaining the nature of the problem. 
For example, in the case of a television that has failed, it is important to detennine if some parts 
of the device are still working. Is the power light on? Is the sound missing, or the picture, or 
both? If the power light is not on, it may be unplugged. If that is not the problem, the next step 
may be to isolate the problem to one part of the system. For instance, the problem may not be the 
TV at all but rather a faulty DVD, which can be tested by inserting a different DVD. A third step 
might be to learn as much as possible about how the system functions, either from an owner’s 
manual or from someone who is familiar with such systems. Troubleshooting is not confined to 
mechanical and electronic systems. Artists, writers, and musicians also encounter problems that 
require troubleshooting. 

Perhaps the most distinctive feature of troubleshooting is coming up with a number of different 
ideas about what may have caused the failure and then using a logical method for narrowing 
down the possible causes with a series of either-or tests, sometimes called a “fault tree,” until the 
source of the problem is discovered. 

When designing technological systems it is important for engineers to consider maintenance 
costs, as people may wish to pay a little more for a product that is less expensive to maintain. 
Similarly, it is important for engineers to anticipate ways in which complicated products and 
systems are likely to break down and to build into the design simple ways to troubleshoot and fix 
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the most common causes of failure. Factors to consider may include maintenance costs, available 
technologies, time until obsolescence, and environmental impacts. 

Key principles in the area of Maintenance and Troubleshooting that all students can be 
expected to understand at increasing levels of sophistication are: 

• Tools and machines must undergo regular maintenance to ensure their proper 
functioning. 

• Troubleshooting is a systematic approach to diagnosing a technological failure. 

• Taking into account the entire life cycle of a product is an important part of designing. 

At the fourth-grade level, students should know that it is important to care for tools and machines 
so they can be used when they are needed. For example, tools should not be left out in the rain, 
and electronic equipment should be handled with care. Students should also know that if 
something does not work as expected, it is possible to find out what the problem is in order to 
decide if the item should be replaced or determine, if possible, how to fix it. They should know 
that some items, such as ballpoint pens, are designed to be disposable, and they should be able to 
discuss the disposal or recycling of such items. 

Eighth graders should be familiar with the concept of maintenance and should understand that 
failure to maintain a device can lead to a malfunction. They should also be able to carry out 
troubleshooting, at least in simple situations. For example, they should be able to safely use tools 
and instruments to diagnose a problem in a device, and they should be able to consult manuals or 
talk to experienced individuals to learn how the device works. They should be able to test 
various ideas for fixing the device. And they should be able to analyze an item’s life cycle and 
discuss the impact of disposing of an item that has reached the end of its useful life. 

By 12th grade, students should know that many devices are designed to operate with high 
efficiency only if they are checked periodically and properly maintained. They should also have 
developed the capability to troubleshoot devices and systems, including those that they may have 
little experience with. Students at this level should also be able to think ahead and to identify and 
document new maintenance procedures so that a malfunction is less likely to occur again. They 
should be able to weigh the costs and benefits of maintaining an existing item versus disposing 
of it and obtaining a newer replacement, with particular attention paid to lessening the 
environmental impact of disposing of obsolescent or nonfunctioning products. 
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Table 2.9 Maintenance and Troubleshooting assessment targets for grades 4, 8, and 12 


D. Maintenance and Troubleshooting 

Fourth graders should recognize that tools and machines need to be cared for and that devices that fail can be fixed 
or replaced. Eighth graders should know that tools and machines must be maintained and be able to use a 
troubleshooting process to diagnose problems in technological systems. Twelfth graders should understand the 
importance of maintenance, be able to analyze malfunctions, and be able to devise ways to reduce future failures. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

D.4.16: It is important to care for 
different tools and machines in 
appropriate ways so that they are 
available to be used when needed. 

Students know that: 

D.8.16: Many different kinds of 
products must undergo regular 
maintenance, including lubrication 
and replacement of parts before they 
fail so as to ensure proper 
functioning. 

Students know that: 

D.12.16: Products and structures of 
various kinds can be redesigned to 
eliminate frequent malfunctions and 
reduce the need for regular 
maintenance. 

Students are able to: 

D.4.17: Change one aspect of a 
machine or tool at a time to discover 
why it is not working. Retest after 
each change has been made. 

Students are able to: 

D.8.17: Diagnose a problem in a 
technological device using a logical 
process of troubleshooting. Develop 
and test various ideas for fixing it. 

Students are able to: 

D.12.17: Analyze a system 
malfunction using logical reasoning 
(such as a fault tree) and appropriate 
diagnostic tools and instruments. 
Devise strategies and recommend 
tools for fixing the problem. 

D.4.18: Identify the cause of failure 
in a simple system and suggest ways 
that failure could be avoided in the 
future. 

D.8.18: Modify a moderately 
complicated system so that it is less 
likely to fail. Predict the extent to 
which these modifications will 
affect the productivity of the system. 

D.12.18: Analyze a complicated 
system to identify ways that it might 
fail in the future. Identify the most 
likely failure points and recommend 
safeguards to avoid future failures. 

D.4.19: Recognize that all products 
have a life cycle, starting with raw 
materials and ending with disposal 
or recycling. 

D.8.19: Trace the life cycle of a 
repairable product from inception to 
disposal or recycling in order to 
determine the product’s 
environmental impact. 

D.12.19: Taking into account costs 
and current trends in technology, 
identify how long a product should 
be maintained and repaired and how 
it might be redesigned to lessen 
negative environmental impacts. 
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Information and Communication Technology (ICT) 


The integration of Information and Communication 
Technologies (ICT) into every sphere of 
contemporary life has had profound implications for 
how people learn in school, solve practical 
problems, and function in the workplace. 

Networked computing and communications 
technologies and media have become essential tools 
of practically every profession and trade, including 
those of lawyers, doctors, artists, historians, 
electricians, mechanics, and salespersons. These 
devices make it possible to redistribute learning and 
work experiences over time and space. Tools 
employed in various professions and trades, such as 
word processors, spreadsheets, audio, video, and 
photo editing tools, models, visualizations, and mobile wireless devices, are, in turn, being put to 
work in the study of core school subjects. Students are able to connect, access, and communicate 
with the wider world in ways that were unimaginable just a few years ago and that are 
continually changing. Particularly relevant for this framework is the fact that virtually all efforts 
to improve or create new technologies involve the use of ICT tools. And for many years to come, 
such novel technologies, computer-based and otherwise, will continue to bring about new 
approaches to education, work, entertainment, and daily life. 

As the term is used in this framework, ICT includes a wide variety of technologies, including 
computers and software learning tools, networking systems and protocols, hand-held digital 
devices, digital cameras and camcorders, and other technologies, including those not yet 
developed, for accessing, managing, creating, and communicating information. 

Although ICT is just one among many different types of technologies, it has achieved a special 
prominence in technology and engineering literacy because familiarity and facility with ICT is 
essential in virtually every profession in modern society, and its importance is expected to grow 
over the coming decades. A wide variety of ICT tools are routinely used in schools, the 
workplace, and homes. Rapidly evolving learning tools such as computers, online media, 
telecommunications, and networked technologies are becoming powerful supports for 
communities of learning and practice. Moving far beyond traditional text-based communication 
methods, the common language of global information sources and communication has broadened 
to include vast collections of images, music, video, and other media. Computers, networks, 
telecommunications, and media support collaboration, expression, and dissemination ranging 
from data organization and analysis, research, scholarship, and the arts to peer interactions. Ever- 
shrinking computer chips are put to work in a collection of devices that seems to be growing 
exponentially and that, at present, includes cellphones, digital assistants, media players, and 
geographical information systems, among a host of other devices. 

Students should be aware of these devices and know how and when to use them. They must have 
mastered a wide range of ICT tools in common use, and they must have the confidence and 
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capability to learn to use new ICT tools as they become available. Although students are not 
expected to understand the inner workings of these devices, they should have enough of an 
understanding of the principles underlying them to appreciate the basics of how they work. Five 
subareas of ICT literacy have been identified for assessment: 

A. Construction and Exchange of Ideas and Solutions; 

B. Information Research; 

C. Investigation of Problems; 

D. Acknowledgment of Ideas and Infonnation; and 

E. Selection and Use of Digital Tools. 

Each of the above subareas relates to one of the broad categories included in the National 
Educational Technology Standards for Students (NETS*S), and standards and frameworks 
developed by the Partnership for 21st Century Skills, the American Association of School 
Librarians, and the International Technology and Engineering Educators Association. The link 
between these subareas and the N ETS • S and the Framework for 21st Century Learning is 
outlined in appendix E. 

A. Construction and Exchange of Ideas and Solutions 

Year after year, information and communication technologies challenge people to think, learn, 
and work in ways that were unimaginable only a short time ago and, as a result, enhance 
communication and collaboration among individuals, groups, and organizations. Several lines of 
research have shown that teams are more productive than individuals for generating solutions to 
many kinds of problems, provided team members are effective collaborators. The findings 
extend to teams that collaborate online as well as face to face (Hennessey and Amabile, 2010). 
For schools, this continuing evolution translates into an increasing need to provide students with 
opportunities to develop digital and media communication skills and to collaborate in 
nontraditional learning environments. 

Several recent sets of national standards and many state standards cite effective communication 
skills and the capability to work collaboratively as essential for student success in the 21st 
century. In addition to mastering a set of computer-based skills, students should be able to 
employ a variety of media and technologies in order to communicate ideas, interact with others, 
and present infonnation to multiple audiences. To be effective collaborators, students should be 
able to negotiate team roles and resources, draw on the expertise and strengths of other team 
members and remote experts, monitor progress toward goals, and reflect on and refine team 
processes for achieving goals. Since the assessment will be administered only to individuals, 
tasks involving collaboration with others and sending and receiving communications will need to 
provide virtual, computer-based renditions of remote collaborators. 

Key principles in the area of Construction and Exchange of Ideas and Solutions that all 
students can be expected to understand at increasing levels of sophistication are: 

• Communication and collaboration are affected (in terms of quantity, quality, and results) 
by the choice of digital tools used. 
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• Digital tools offer many options for formal and nonfonnal expression in nearly every 
academic and professional discipline. 

• Teams need people with a variety of skills. 

Fourth-grade students should understand what is expected from members working as part of a 
team and should realize that teams are better than individuals at solving many kinds of problems. 
Students should be able to gather information from various sources and share ideas with a 
specified audience. 

Eighth-grade students should know that communicating always involves understanding the 
audience — the people for whom the message is intended. They should also be able to use 
feedback from others, and provide constructive criticism. 

Twelfth-grade students are expected to have developed a number of effective strategies for 
collaborating with others and improving their teamwork. They should be able to synthesize 
information from different sources and communicate with multiple audiences. 
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Table 2.10 Construction and Exchange of Ideas and Solutions assessment targets for 

grades 4, 8, and 12 


A. Construction and Exchange of Ideas and Solutions 

Fourth-grade students should be able to collaborate and communicate by working with other members of a (virtual) 
team to make decisions and develop presentations using a variety of formats. Eighth-grade students should be able 
to take into account the perspective of different audiences, use a variety of media to create effective messages, and 
modify presentations based on feedback (virtual). Twelfth-grade students should have developed strategies to be 
effective collaborators, should be able to take into account multiple viewpoints, and should be able to synthesize 
information from a variety of sources. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.1: People collaborating as a team 
can often produce a better product 
than people working alone. There 
are common digital tools that can be 
used to facilitate virtual or face-to- 
face collaboration. 

Students know that: 

1.8.1: Collaboration can take many 
forms. Pairs or teams of people can 
work together in the same space or 
at a distance, at the same time or at 
different times, and on creative 
projects or on technical tasks. 
Different communications 
technologies are used to support 
these different forms of 
collaboration. 

Students know that: 

1.12.1: Effective collaboration 
requires careful selection of team 
members, monitoring of progress, 
strategies for reaching agreement 
when there are opposing points of 
view, and iterative improvement of 
collaborative processes. Information 
and communication technologies can 
be used to record and share different 
viewpoints and to collect and 
tabulate the views of groups of 
people. 

Students are able to: 

1.4.2: Utilize input from (virtual, 
that is, computer-generated) 
collaborators and experts or sources 
in the decision-making process to 
design a product or presentation. 

Students are able to: 

1.8.2: Provide feedback to a (virtual) 
collaborator on a product or 
presentation, taking into account the 
other person’s goals and using 
constructive, rather than negative, 
criticism. 

Students are able to: 

1.12.2: Work through a simulation of 
a collaborative process. Negotiate 
team roles and resources, draw upon 
the expertise and strengths of other 
team members and remote experts, 
monitor progress toward goals, and 
reflect on and refine team processes 
for achieving goals. 

1.4.3: Communicate information and 
ideas effectively to an audience in 
order to accomplish a specified 
purpose. 

1.8.3: Communicate information and 
ideas effectively using a variety of 
media, genres, and formats for 
multiple purposes and a variety of 
audiences. 

1.12.3: Synthesize input from 
multiple sources to communicate 
ideas to a variety of audiences using 
various media, genres, and formats. 
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B. Information Research 

Research and infonnation capability is a central ICT skill. In using digital and networking tools 
to find relevant and useful information, students must first be able to formulate a set of questions 
that will guide them in their search, and they must also be capable of synthesizing data from 
multiple sources. Students must be able to fonnulate efficient search strategies and to evaluate 
the credibility of information and data sources. They must extract and save information and data 
that they judge to be relevant to the question at hand. And they must be able to use multiple ICT 
tools to organize, synthesize, and display information and data. 

Key principles in the area of Information Research that all students can be expected to 
understand at increasing levels of sophistication are: 

• Increases in the quantity of information available through electronic means and the ease 
with which knowledge can be published have heightened the need for the verification of 
sources of expertise. 

• Infonnation can be distorted, exaggerated, or otherwise misrepresented. 

• Important strategies for ensuring quality of information include 1) assessing the source of 
information and 2) using multiple sources to verily the information in question. 

• Search strategies and skills are important capabilities in performing effective infonnation 
research. 

Fourth-grade students should be aware of a number of digital and network tools that can be used 
for finding infonnation, and they should be able to use these tools to collect, organize, and 
display data in response to specific questions and to help solve problems. 

Eighth-grade students should be aware of digital and network tools and be able to use them 
efficiently. They should be aware that some of the infonnation they retrieve may be distorted, 
exaggerated, or otherwise misrepresented, and they should be able to identify cases where the 
information is suspect. 

Twelfth-grade students should be able to use advanced search methods and select the best digital 
tools and resources for various purposes. They should also be able to evaluate information for 
timeliness and accuracy, and they should have developed strategies to check the credibility of 
sources. 
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Table 2.11 Information Research assessment targets for grades 4, 8, and 12 


B. Information Research 

Fourth-grade students can use digital and network tools to find information and identify sources that may be biased 
in some way. Eighth-grade students are able to use digital resources to find information and also to recognize when 
information may be distorted, exaggerated, or otherwise misrepresented. Twelfth-grade students can use advanced 
search methods and select the best digital tools and resources for various purposes, can evaluate information for 
timeliness and accuracy, and can check the credibility of sources. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.4: Digital and network tools and 
media resources are helpful for 
answering questions, but they can 
sometimes be biased or wrong. 

Students know that: 

1.8.4: Increases in the quantity of 
information available through 
electronic means and the ease by 
which knowledge can be published 
have heightened the need to check 
sources for possible distortion, 
exaggeration, or misrepresentation. 

Students know that: 

1.12.4: Advanced search techniques 
can be used with digital and network 
tools and media resources to locate 
information and to check the 
credibility and expertise of sources. 

Students are able to: 

1.4.5: Use digital and network tools 
and media resources to collect, 
organize, and display data in order 
to answer questions and solve 
problems. 

Students are able to: 

1.8.5: Select and use appropriate 
digital and network tools and media 
resources to collect, organize, 
analyze, and display supporting data 
to answer questions and test 
hypotheses. 

Students are able to: 

1.12.5: Select digital and network 
tools and media resources to gather 
information and data on a practical 
task, and justify choices based on the 
tools’ efficiency and effectiveness 
for a given purpose. 

1.4.6: Search media and digital 
sources on a community issue and 
identify sources that may be biased. 

1.8.6: Search media and digital 
resources on a community or world 
issue and identify specific examples 
of distortion, exaggeration, or 
misrepresentation of information. 

1.12.6: Search media and digital 
resources on a community or world 
issue and evaluate the timeliness and 
accuracy of the information as well 
as the credibility of the source. 
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C. Investigation of Problems 

In addition to helping students find information, digital tools are widely used in core school 
subjects to support students’ critical thinking, problem-solving, and decision-making. In 
language arts courses, for example, students use online graphic organizers, word processors, and 
media as they read, analyze, and draw conclusions about various texts. They launch discussions 
on wikis to stimulate a rich consideration of topics prior to class time and to give students who 
are less confident in face-to-face situations the opportunity to be major contributors. In social 
science courses, students use databases and spreadsheets to create tables and graphs as they 
analyze and compare population densities in different historical periods. In science and 
mathematics, students use spreadsheets, visualization and modeling tools, digital probeware, and 
presentation tools to gather and interpret data on science and health issues. 

This subarea is closely related to the previous one, Information Research, but the focus here is on 
problem-solving and critical thinking as opposed to searching for information. In practice, of 
course, the two sets of skills will often be applied at the same time, but for the purpose of 
assessment it is useful to keep them separate. 

Schools are society’s means of preparing students for the world, and so many of the ways that 
students use digital tools reflect the way that professionals use similar tools to solve various 
practical problems, such as environmental issues, political conflicts, and economic challenges. In 
these cases, digital tools may be used to present the challenge scenario; to guide students in 
formulating the requirements of the challenge to be addressed; and to allow students to ask and 
answer significant questions, to exchange views with other students, sometimes in other cities or 
countries, to collect and analyze data, and then to develop and test various solutions through 
simulations. Other uses of digital tools in schools involve practical applications designed to 
prepare students for the responsibilities of adulthood. 

Key principles in the area of Investigation of Problems that all students can be expected to 
understand at increasing levels of sophistication are: 

• Digital tools can be very helpful in generating ideas and solving problems in academic 
subjects as well as in researching practical problems. 

• Digital models can be used to create simulations and test solutions. 

• Digital tools can be used to conduct experiments and investigate practical problems. 

Fourth-grade students should be able to use a variety of information and communication 
technologies to investigate a local or otherwise familiar issue and to generate, present, and 
advocate for possible solutions. They should also be able to use digital tools to test hypotheses in 
various subject areas and to build models of simple systems. 

Eighth-grade students should be able to use digital tools to identify and research a global issue 
and to identify and compare different possible solutions. They should also be able to use digital 
tools in testing hypotheses of moderate complexity in various subject areas in which they gather, 
analyze, and display data and draw conclusions. They should also be able to explore authentic 
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issues by building models and conducting simulations in which they vary certain quantities to 
test “what if’ scenarios. 

Twelfth-grade students should be able to use digital tools to research global issues and to fully 
investigate the pros and cons of different approaches. They should be able to design and conduct 
complex investigations in various subject areas using a variety of digital tools to collect, analyze, 
and display infonnation and be able to explain the rationale for the approaches they used in 
designing the investigation as well as the implications of the results. Twelfth-grade students 
should also be able to conduct simulations, draw conclusions based on the results, and critique 
the conclusions based on the adequacy of the model to represent the actual problem situation. 
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Table 2.12 Investigation of Problems assessment targets for grades 4, 8, and 12 


C. Investigation of Problems 

Fourth-grade students are able to use digital tools to investigate local issues, test hypotheses, and build models. 
Eighth-grade students are able to use digital tools to investigate alternative solutions to global issues, test moderately 
complex hypotheses, build models, and conduct simulations. Twelfth-grade students can conduct more sophisticated 
investigations and simulations as well as recognize their limitations. For all levels the focus is on types of hardware 
and software rather than on use of particular hardware or software products. 

Grade 4 

Grade 8 

Grade 12 

Students are able to: 

1.4.7: Use digital tools and resources 
to identify and investigate a local 
issue and generate possible 
solutions. 

Students are able to: 

1.8.7: Use digital tools to identify a 
global issue and investigate possible 
solutions. Select and present the 
most promising sustainable solution. 

Students are able to: 

1.12.7: Use digital tools and 
resources to identify a complicated 
global issue and develop a 
systematic plan of investigation. 
Present findings in terms of pros and 
cons of two or more innovative 
sustainable solutions. 

1.4.8: Use digital tools to test simple 
hypotheses in various subject areas. 

1.8.8: Use digital tools to gather and 
display data in order to test 
hypotheses of moderate complexity 
in various subject areas. Draw and 
report conclusions consistent with 
observations. 

1.12.8: Use digital tools to collect, 
analyze, and display data in order to 
design and conduct complicated 
investigations in various subject 
areas. Explain rationale for the 
design and justify conclusions based 
on observed patterns in the data. 

1.4.9: Use digital models to describe 
how parts of a whole interact with 
each other in a model of a system. 

1.8.9: Use a digital model of a 
system to conduct a simulation. 
Explain how changes in the model 
result in different outcomes. 

1.12.9: Having conducted a 
simulation of a system using a 
digital model, draw conclusions 
about the system, or propose 
possible solutions to a problem or 
ways to reach a goal based on 
outcomes of the simulation. Critique 
the conclusions based on the 
adequacy of the model. 
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D. Acknowledgment of Ideas and Information 

Digital citizenship is an essential element of technology and engineering literacy. As rapid 
technological advances have increased people’s capacity to access and share information 
anytime and anywhere around the globe, there is increasing concern about the misuse and abuse 
of information. Some of the ethical and legal concerns were described under Technology and 
Society and include worries about such issues as providing false information, invading people’s 
privacy, “hacking” into secure networks, and using ICT tools for industrial espionage. There is 
therefore a certain amount of overlap between that subarea and this one, but this subarea is 
focused specifically on an especially important category of ethical issues: the appropriate use of 
intellectual property in the context of digital media. 

For many students the first opportunity to leam about the ethical implications of intellectual 
property appears in discussions about classroom cheating, in which a student looks at someone 
else’s test paper and writes down answers and ideas as his or her own. At the highest levels of 
academia, this practice is known as “plagiarism,” and allegations of plagiarism can lead to 
criminal as well as ethical sanctions. On the other hand, it is not cheating to incorporate other 
people’s ideas as long as credit for the source of the ideas is given at the time they are used. It is 
therefore essential that students know the conventional methods for appropriately crediting 
others’ ideas, words, and images, both orally and in the form of writing and other media. 

A closely related issue is the use and misuse of copyrighted material. Even at the elementary 
level it has become so easy to copy and share digital infonnation that children need to understand 
the importance of respecting copyrighted materials so that they will be more likely as adults to 
continue to honor intellectual property rights and laws protecting patents, trademarks, copyrights, 
music, and video. Although technological safeguards may be developed in future years, 
individual respect for the intellectual property of others will continue to be an important ethical 
imperative. 

Key principles in the area of Acknowledgment of Ideas and Information that all students can 
be expected to understand at increasing levels of sophistication are: 

• Copyright laws and policies are designed to protect intellectual property. 

• Fair use guidelines are designed to support the use of copyrighted materials for academic 
purposes and for journalism and other forms of writing and commentary. 

• There are multiple guiding principles (laws, policies, and guidelines) that govern the use of 
ideas and information. 

Fourth-grade students should understand that it is permissible to use others’ ideas as long as 
appropriate credit is given. This ethical guideline that one should give credit where it is due 
holds true not just for tests and homework, but also even in everyday conversation. They should 
also know that copyrighted materials cannot be shared freely. 

Eighth-grade students should be aware of general principles concerning the use of other people’s 
ideas and know that these principles are the basis for such things as school rules and federal laws 
governing such use. They should know about the limits of fair use of verbatim quotes and how to 
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cite sources in papers or other media productions. They should understand the importance of 
giving appropriate credit for others’ ideas and contributions, and they should know how to give 
such credit. 

Twelfth-grade students should understand the fundamental reasons for intellectual property laws 
and should know acceptable practices for citing sources when incorporating ideas, quotes, and 
images into their own work. 
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Table 2.13 Acknowledgment of Ideas and Information assessment targets for 

grades 4, 8, and 12 


D. Acknowledgment of Ideas and Information 

Fourth-grade students exhibit digital citizenship by understanding that it is permissible to use others’ ideas as long as 
appropriate credit is given but that copyrighted materials cannot be shared freely. Eighth-grade students should be 
aware of and comply with laws and ethical guidelines for incorporating ideas, text, and images into their own work. 
Twelfth-grade students should understand the reasons for protecting intellectual property and demonstrate 
responsible and ethical behaviors when using ideas, quotes, and images from others. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.10: It is allowable to use other 
people’s ideas in one’s own work 
provided that proper credit is given 
to the original source, whether 
information is shared in person or 
through ICT media. 

Students know that: 

1.8.10: Style guides provide detailed 
examples for how to give 
appropriate credit to others when 
incorporating their ideas, text, or 
images in one’s own work. 

Students know that: 

1.12.10: Legal requirements 
governing the use of copyrighted 
information and ethical guidelines 
for appropriate citations are intended 
to protect intellectual property. 

Students are able to: 

1.4.11: Identify or provide examples 
demonstrating respect for 
copyrighted material, such as 
resisting the request from a friend to 
copy a song from a CD or placing 
copyrighted material online. 

Students are able to: 

1.8.11: Identify or provide examples 
of fair use practices that apply 
appropriate citation of sources when 
using information from books or 
digital resources. 

Students are able to: 

1.12.11: Identify or provide 
examples of responsible and ethical 
behavior that follow the letter and 
spirit of current laws concerning 
personal and commercial uses of 
copyrighted material as well as 
accepted ethical practices when 
using verbatim quotes, images, or 
ideas generated by others. 


2-43 
















Technology and Engineering Literacy Framework for the 2014 NAEP 


E. Selection and Use of Digital Tools 

Until recently, classroom uses of technology tended to focus almost exclusively on helping 
students become competent users of the technology itself. Educators today have come to 
recognize that how technology is used as a tool for learning is at least as important — if not more 
so — than simply how to use technology. Nonetheless, students still require basic operational 
skills and concepts to be effective users of technology for learning. These skills include the 
capabilities to select and use the appropriate tools, to use those tools to complete tasks effectively 
and productively, and to apply current knowledge about technology to leam how to use new 
technologies as they become available. 

Key principles in the area of Selection and Use of Digital Tools that all students can be 
expected to understand at increasing levels of sophistication are: 

• Knowledge about the common uses of readily available digital tools supports effective tool 
selection. 

• A fundamental aspect of technology and engineering literacy is the possession of 
foundational ICT skills in the use of common productivity tools. 

Fourth-grade students should know that different digital tools have different purposes. They 
should also be able to use a variety of digital tools that are appropriate for their age level. For 
example, they should be reasonably competent in using digital tools for creating documents and 
images, for solving problems, and for collecting and organizing infonnation. 

Eighth-grade students should be familiar with different types of digital tools and be able to move 
easily from one type of tool to another — for example, creating a document or image with one 
tool and then using a second tool to communicate the result to someone at a distant location. 

They should be able to select and use effectively a number of tools for different purposes. 

Twelfth-grade students should be competent in the use of a broad variety of digital tools and be 
able to explain why some tools are more effective than others that were designed to serve the 
same purpose, based on the features of the individual tools. 
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Table 2.14 Selection and Use of Digital Tools assessment targets for grades 4, 8, and 12 


E. Selection and Use of Digital Tools 

Fourth-grade students know that different digital tools have different purposes and are able to use a number of 
different tools. Eighth-grade students can categorize digital tools by function and can select appropriate tools and 
demonstrate effective use of the tools for different purposes. Twelfth-grade students are competent in the use of a 
broad variety of digital tools and can justify why certain tools are chosen over others that might accomplish the same 
task, by referencing specific features. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.12: Different digital tools have 
different purposes. 

Students know that: 

1.8.12: Certain digital tools are 
appropriate for gathering, 
organizing, analyzing, and 
presenting information, while other 
kinds of tools are appropriate for 
creating text, visualizations, and 
models and for communicating with 
others. 

Students know that: 

1.12.12: A variety of digital tools 
exist for a given purpose. The tools 
differ in features, capacities, 
operating modes, and style. 
Knowledge about many different 
ICT tools is helpful in selecting the 
best tool for a given task. 

Students are able to: 

1.4.13: Use digital tools (appropriate 
for fourth-grade students) 
effectively for different purposes, 
such as searching, organizing, and 
presenting information. 

Students are able to: 

1.8.13: Use appropriate digital tools 
to accomplish a variety of tasks, 
including gathering, analyzing, and 
presenting information as well as 
creating text, visualizations, and 
models and communicating with 
others. 

Students are able to: 

1.12.13: Demonstrate the capability 
to use a variety of digital tools to 
accomplish a task or develop a 
solution for a practical problem. 
Justify the choice of tools, explain 
why other tools were not used based 
on specific features of the tools, and 
summarize the results. 
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Conclusion 

This chapter has described in detail the knowledge and capabilities related to technology that will 
be assessed on the 2014 NAEP Technology and Engineering Literacy Assessment. While these 
assessment targets represent an important component of technology and engineering literacy, 
they are not intended to limit the scope of concepts and capabilities that could be addressed in 
curricula. As NAEP is an on-demand test, important knowledge and capabilities that should fonn 
a part of every students’ education but are difficult to assess have been excluded from this 
framework. 

Although chapter two is central to the design of the NAEP Technology and Engineering Literacy 
Assessment, it is not sufficient in itself to describe the kinds of reasoning to be expected from 
students, the context or subject matter that will be used to construct test items, or the overall 
shape of the entire assessment. These issues are taken up in chapters three and four. 
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Chapter Three: Practices and Contexts for 
Technology and Engineering Literacy 


Introduction 

Chapter two described the assessment targets for technology and engineering literacy, both 
knowledge and skills that NAEP will assess. This chapter describes three generalizable practices 
that represent the kinds of thinking and application that will be expected of students across the 
three major assessment areas. The chapter also explains the contexts, or situations and types of 
problems, in which assessment tasks and items will be set. These three elements and their 
relationships are portrayed in figure 2. 



Figure 2 - Elements of the NAEP Technology and Engineering Literacy Assessment 


As indicated in figure 2, the practices expected of students are general, crosscutting reasoning 
processes that students must use to show that they understand and can use their technological 
knowledge and skills. The contexts in which technology and engineering literacy tasks and items 
appear will include typical issues, problems, and goals that students might encounter in school or 
practical situations. The particular knowledge and skills that are the targets of the assessment lie 
in the three assessment areas. Together, the assessment targets, practices, and contexts provide a 
structure for the generation of tasks and items. This chapter describes the practices and contexts 
in some detail and provides examples of the types of tasks and items that result when these three 
elements are combined. 
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Practices 

Practices contribute to the framework by articulating the general kinds of thinking and reasoning 
that students are expected to demonstrate when responding to assessment tasks and items. The 
framework specifies three generic kinds of practices that apply across the three assessment areas: 
(1) understanding technological principles; (2) developing solutions and achieving goals; and (3) 
communicating and collaborating. 


Building on the pyramid of major assessment 
areas introduced in chapter two, the practices 
are distributed around a circle in which the 
pyramid sits. As critical components of 
technological literacy, these crosscutting 
practices are applied across all three major 
assessment areas. For example, 
communicating effectively and collaborating 
with others are necessary skills for 
understanding the effects of technology on the 
natural world, designing an engineering 
solution to a technological problem, and 
achieving a goal using information and 
communication technologies. 



The Science Framework for the 2009 National Assessment of Educational Progress served as the 
primary source for these practices. Developers of the science framework had examined the 
sections on Science and Technology and the Designed World from the National Science 
Education Standards as well as Benchmarks for Scientific Literacy and cognitive research on 
science learning. Learning research has shown that these three kinds of cognitive processes 
represent how all individuals build their content knowledge and understanding in a subject area 
and how they develop strategies for using their knowledge in their thinking, reasoning, and 
application to new situations. To create the practices for the NAEP Technology and Engineering 
Literacy Framework, the crosscutting practices from the Science Framework for the 2009 NAEP 
were modified so that they applied to processes relevant to technological literacy, primarily by 
indicating that technology and engineering literacy depends on having and using knowledge 
about technologies to reason, develop solutions, communicate, and collaborate. In addition, the 
national, international, and state technology and engineering literacy frameworks cited in chapter 
two were used as reference points. 

Although the practices are related and not independent of each other, classifying the assessment 
targets in chapter two according to the three crosscutting practices will help developers produce a 
range of rich and challenging assessment tasks and items. A brief description of each of these 
three practices is offered below. 

Understanding Technological Principles focuses on students’ knowledge and understanding of 
technology and their capability to think and reason with that knowledge. This practice ranges 
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from the knowledge of simple declarative facts and concepts to higher-level reasoning about 
facts, concepts, and principles and their interrelationships. Students should be able to call 
on their recognition and understanding of technological principles to explain features and 
functions of technologies and systems, how components fit together, and to make predictions, 
comparisons, and evaluations. The assessment targets that elicit this practice require students to 
identify examples, explain, describe, analyze, compare, relate, and represent the technological 
principles specified in chapter two. This practice also includes understanding the relationships 
among components of systems and interactive processes. 

Developing Solutions and Achieving Goals refers to students’ systematic application of 
technological knowledge, tools, and skills to address problems and achieve goals presented in 
societal, design, curriculum, and realistic contexts. This practice includes both procedural and 
strategic capabilities — knowing how to apply simple steps and use technological tools to address 
authentic tasks, as well as when and where to apply the tools and design and problem-solving 
strategies. This practice draws on the previous practice — to understand technological 
principles — and adds the dimension of applying this knowledge to solve problems and achieve 
goals. This practice involves using fundamental problem-solving processes such as planning, 
monitoring, evaluating, and revising, and how these generic problem-solving strategies can be 
employed in the three assessment areas. It may engage students in analyzing goals, planning, 
designing, and implementing as well as in iteratively revising and evaluating possible solutions 
to meet the requirements of a problem or to achieve a goal. For the NAEP Technology and 
Engineering Literacy Assessment, a distinguishing feature of this practice is that the students 
respond to questions and tasks during the process of solving a multistage problem or working 
through how best to achieve a goal, using their understanding of technological principles to do 
so. 


Communicating and Collaborating centers on students’ capabilities to use contemporary 
technologies to communicate for a variety of purposes and in a variety of ways, working 
individually or in teams. In the three major assessment areas, in order to address societal issues, 
solve problems, achieve goals, and design processes and products, students must develop 
representations and share ideas, designs, data, explanations, models, arguments, and 
presentations. Effective teamwork and collaboration with peers and experts who are either 
present or in another location are important skills that can help students achieve their goals. In 
the assessment, collaboration tasks will engage individual students with virtual (computer- 
generated) peers and experts. Communication and collaboration are critical, crosscutting 
practices in all subject domains. For technology and engineering literacy, these practices are 
distinguished by students’ facility with a range of technologies to communicate and collaborate. 

Table 3.1 presents generic examples of how these 3 practices can be used to classify targets in 
the 3 major assessment areas. These are sample ideas for items and tasks and will not be used in 
the actual assessment. It should again be noted that the boundaries between the practices are not 
entirely distinct, but referring to these three practices can be helpful in the development of tasks 
and items and the interpretation of student performance for a range of cognitive demands. 
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Table 3.1 Classification of types of assessment targets in the three major assessment areas 
according to the practices for technology and engineering literacy 



Technology and Society 

Design and Systems 

Information and 
Communication 
Technology (ICT) 

Understanding 

Technological 

Principles 

Analyze advantages and 
disadvantages of an existing 
technology 

Explain costs and benefits 
Compare effects of two 
technologies on individuals 
Propose solutions and alternatives 
Predict consequences of a 
technology 

Select among alternatives 

Describe features of a system 
or process 

Identify examples of a system 
or process 

Explain the properties of 
different materials that 
determine which is suitable to 
use for a given application or 
product 

Analyze a need 
Classify the elements of a 
system 

Describe features and 
functions of ICT tools 
Explain how parts of a whole 
interact 

Analyze and compare relevant 
features 

Critique a process or outcome 
Evaluate examples of effective 
resolution of opposing points 
of view 

Justify tool choice for a given 
purpose 

Developing 
Solutions and 
Achieving 
Goals 

Select appropriate technology to 
solve a societal problem 
Develop a plan to investigate an 
issue 

Gather and Organize data and 
information 

Analyze and Compare 

advantages and disadvantages of a 
proposed solution 
Investigate environmental and 
economic impacts of a proposed 
solution 

Evaluate trade-offs and impacts 
of a proposed solution 

Design and Build a product 
using appropriate processes and 
materials 

Develop forecasting techniques 
Construct and Test a model or 
prototype 

Produce an alternative design 
or product 
Evaluate trade-offs 
Determine how to meet a need 
by choosing resources required 
to meet or satisfy that need 
Plan for durability 
Troubleshoot malfunctions 

Select and Use appropriate 
tools to achieve a goal 
Search media and digital 
resources 

Evaluate credibility and 
solutions 

Propose and Implement 

strategies 

Predict outcomes of a 
proposed approach 
Plan research and presentations 
Organize data and information 
Transform from one 
representational form to 
another 

Conduct experiments using 
digital tools and simulations 

Communicating 

and 

Collaborating 

Present innovative, sustainable 
solutions 

Represent alternative analyses 
and solutions 

Display positive and negative 
consequences using data and 
media 

Compose a multimedia 
presentation 

Produce an accurate timeline of a 
technological development 
Delegate team assignments 
Exchange data and information 
with virtual peers and experts 

Display design ideas using 
models and blueprints 
Use a variety of media and 
formats to communicate data, 
information, and ideas 
Exhibit design of a prototype 
Represent data in graphs, 
tables, and models 
Organize, Monitor, and 
Evaluate the effectiveness of 
design teams 

Request input from virtual 
experts and peers 

Provide and Integrate 

feedback 

Plan delegation of tasks among 
team members 
Provide and Integrate 
feedback from virtual peers and 
experts to make changes in a 
presentation 
Critique presentations 
Express historical issues in a 
multimedia presentation 
Argue from an opposing point 
of view 

Explain to a specified 
audience how something works 
Address multiple audiences 
Synthesize data and points of 
view 
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Examples of Practices Applied in Each of the Assessment Areas 

The following sections describe how the three practices of Understanding Technological 
Principles, Developing Solutions and Achieving Goals, and Communicating and Collaborating 
can be used to classify the general types of thinking and reasoning intended by the assessment 
targets in the three major assessment areas of Technology and Society, Design and Systems, and 
Infonnation and Communication Technology (ICT). 

Practices Applied in Technology and Society 

Assessment targets in the area of Technology and Society are concerned with the effects of 
technology on human society, the natural world, and the world of information and knowledge as 
well as with issues of ethics, equity, and responsibility. 

Understanding Technological Principles 

To provide evidence that they understand principles in these three subareas, students could be 
asked to perfonn a variety of tasks, such as recognizing examples of the effects of technologies; 
identifying examples of ethical and equity issues; describing local and global effects of 
technologies; explaining the effects of rapidly changing technologies on knowledge creation, 
access, and management; analyzing beneficial and negative impacts; recognizing examples of 
responsible, ethical uses of technologies; comparing costs and benefits of technologies; 
predicting potential impacts on society and the environment; and explaining the relationships 
among technologies. 

Developing Solutions and Achieving Goals 

Students must use their understanding of the technological principles for Technology and Society 
specified in chapter two to apply that knowledge as they address novel issues and problems. To 
demonstrate their capacity to address issues and problems in the assessment area of Technology 
and Society, students could be asked, for example, to develop alternative proposals for a new 
technology based on an analysis of potential positive and negative impacts. Problem-solving 
practices could be demonstrated in a series of tasks and items involved in analyzing the uses of 
the new technology, gathering data and infonnation on its impacts, analyzing the data, 
interpreting results, and evaluating alternatives. 

Communicating and Collaborating 

To communicate and collaborate with others (virtual others in the assessment) in the course to 
respond to issues, students must draw on their understanding of the technological principles 
specified in chapter two and apply their knowledge as they work through given problems and 
issues. For example, to address issues in Technology and Society, students can use a variety of 
modalities to represent and exchange data, ideas, and arguments about the advantages and 
disadvantages of technologies. Students can collaborate (virtually) to form teams that will gather 
and integrate information about the potential impacts of a technology on human society or the 
natural world. Students can evaluate the qualifications, credibility, and objectivity of virtual 
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experts. Tasks can require students to demonstrate their capability to interact, collaborate, and 
contribute to work as a team. Students can use various media and representations to share their 
analyses and recommendations. 

Table 3.2 provides some examples of how the 3 practices can be applied to assessment targets 
for Technology and Society to generate tasks and items at the middle school level. The key 
principles and targets were selected from tables 2.2-2. 5 in chapter two. These are sample ideas 
for items and tasks and will not be used in the actual assessment. Simpler tasks could be 
developed for grade 4 targets and more complex tasks could be developed for grade 12 targets. 
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Table 3.2 Examples of grade 8 tasks representing practices in each subarea of 

Technology and Society 



A. Interaction of 
Technology and 
Humans 

B. Effects of 
Technology on the 
Natural World 

C. Impacts on the 
World of 
Information and 
Knowledge 

D. Ethics, Equity, 
and Responsibility 

\ Selected 

\ Principles 

Practices \ 

The relationship 
between technology 
and society is 
reciprocal. Society 
drives technological 
change, while 
changing 

technologies in turn 
shape society. 

Reusing, recycling, 
and using fewer 
resources can reduce 
environmental 
impacts. 

Information 
technology is 
evolving rapidly, 
enabling ever- 
increasing amounts of 
information and data 
to be stored, 
managed, enhanced, 
analyzed, and 
accessed through a 
wide array of devices 
in various media 
formats. 

Technology by itself 
is neither good nor 
bad, but its use may 
affect others. 

Understanding 

Technological 

Principles 

Explain what factors 
need to go into a 
decision to change 
the use of a river and 
identify possible 
consequences of 
doing so. 

Identify and provide a 
rationale for 
appropriate and 
inappropriate 
procedures for 
disposing of 
electronic devices. 

Compare the impact 
of geographical 
information systems 
and 14th-century 
maps on people’s 
capability to explore 
new territory. 

What are the positive 
and negative 
consequences of the 
predicted change 
from print to digital 
news? 

Developing 
Solutions and 
Achieving Goals 

The community has 
decided to implement 
a new wind turbine 
system. Design an 
investigation into the 
impact on the 
community. 

Given a specific 
consumer electronics 
product such as a 
cellular telephone, 
design a new way to 
increase its 
appropriate disposal. 

Use a simulation to 
test the adequacy of 
exit routes for 
evacuating residents 
of a mountain town 
during a wildfire. 

What processes and 
digital tools might the 
city council put into 
place in order to 
make sure all citizens 
have a say? 

Communicating and 
Collaborating 

Collaborate with 
engineers and urban 
planners (virtual) 
through a website to 
collect and 
communicate data 
about the effects of a 
wind turbine system 
on the community. 

Organize a campaign 
with a virtual team to 
inform the public of 
the dangers of 
improper disposal of 
consumer electronic 
products. 

Present a set of 
images from two 
artists of the period 
that represent 
different perspectives 
on a major event. 

Debate with a virtual 
team member the 
privacy and safety 
issues involved in 
establishing 
international projects. 
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Illustrative Tasks and Items 


The following examples represent three types of tasks and items that could be used to assess 
targets related to the Technology and Society assessment area. The first example is a scenario- 
based set of items, and the next two are familiar, conventional constructed and selected-response 
items. 

Scenario-Based Item Sets 

Items of this type present students with a problem or goal set within a broader context. The 
example below calls on the student to employ the practice of Developing Solutions and 
Achieving Goals. In this task, the student must use ICT tools to analyze the impacts of 
technology on human society. This example employs a computer-based interactive format in 
which students search for information concerning video games and violence and then use word- 
processing, spreadsheet, and Web-based tools to develop a PowerPoint presentation. The 
PowerPoint presentation calls on the crosscutting practice of Communicating and Collaborating. 


Context: Human Society 
Topic: Video Games and Violence 
Target Level: Grade 12 


Video Games 
and Violence 


Tasks 




Time remaining 
24 minute(s) 



TASK DETAILS 

You are going to create a slide show about the relationship between Video 
Games and Violence to present to your class. 

You should use no more than five slides. 

Your slide show will be assessed on: 


• the ideas and information you include; 

• the way you organise the information; 

• the design of the presentation; and 

• your use of the software features. 


This is the big task for this module. 

After this big task you will have one small task to complete but this should 
only take one minute. 

You should spend most of your remaining time on this big task. 

Before you begin this task you will watch a demonstration of how to do it. 


Click on 'I've finished' to watch the demonstration. 


i I’ve jr4f 
finished ^ 
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(Source: Australian Ministerial Council for Education, Early Childhood Development and Youth 
Affairs [MCEECDYA], 2008) 
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Conventional Items 

These items will ask for students to select or construct their answers. The examples below are of 
the familiar constructed- and selected-response formats. 

Constructed Response 

This task illustrates students’ skill in using simulations in a problem-solving activity. The student 
constructs a response by manipulating force arrows in a simulation-based scenario in which an 
emergency rescue truck must deal with various problems along a fire road in a forest. The 
student constructs text-based explanations of the balanced and unbalanced forces acting on the 
truck. While designed to be used as a middle school science item about force and motion, such a 
simulation could be adapted for the NAEP Technology and Engineering Literacy Assessment to 
study how the design of the technological system (transportation) affects the environment 
positively by making it possible to contain forest fires and rescue people and also negatively 
because of the cutting of trees and the disruption of wildlife habitat. This task also illustrates an 
interactive item in which students can manipulate different factors in a simulated environment 
and then provide responses to questions related to the scenario. For the NAEP Technology and 
Engineering Literacy Assessment, a series of items could call on the practices of Understanding 
Technological Principles (for example, describing positive and negative effects of roads in the 
forest) and Developing Solutions and Achieving Goals (for example, designing a fire road that 
would not cross wildlife migration corridors). 


Context: Transportation 
Topic: Emergency Rescue 
Target Level: Grade 8 



(Source: Quellmalz, Timms, and Buckley, 2009) 
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Selected Response — Multiple Choice 

In this conventional multiple-choice item, the student selects an answer that describes the effect 
of fossil fuel on the natural environment, employing the practice of Understanding Technological 
Principles to analyze positive and negative effects. This item assesses student understanding of 
the interaction of a technology (energy production) and climate change. The combination of the 
two can be thought of as a system, with each affecting the other, and the item below also assesses 
a student’s system thinking, albeit on a relatively simple level. Similar items in the NAEP 
Technology and Engineering Literacy Assessment could focus on the effects of technology on 
the environment. 


Context: Natural World 
Topic: Climate Change 
Target Level: Grade 8 

The burning of fossil fuels has increased the carbon dioxide content of the 
atmosphere. What is a possible effect that the increased amount of carbon 
dioxide is likely to have on our planet? 

(a) A warmer climate 

(5) A cooler climate 

(c) Lower relative humidity 

(g) More ozone in the atmosphere 

(Source: Trends in International Mathematics and Science Study, 2003) 


Practices Applied in Design and Systems 

Assessment targets in the area of Design and Systems relate to the nature of technology, the 
engineering design process, maintenance and troubleshooting, and systems thinking. The 
sections below describe how the three crosscutting practices apply to targets in the area of 
Design and Systems. 

Understanding Technological Principles 

Technological principles for Design and Systems in chapter two specify for this assessment the 
core understandings that students should have about the different types of technologies, 
processes for designing technologies, approaches to preventing failures, and how components of 
technological systems interrelate. The practices for Understanding Technological Principles in 
these areas could, for example, ask students to draw on their knowledge to identify examples of 
technologies, components of the design process, components of a system, or maintenance and 
troubleshooting methods. Students could be asked to explain the relationship among technologies 
in a system, analyze the components of a system, recognize design constraints, or evaluate 
alternative representations of a system. 
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Developing Solutions and Achieving Goals 

Problem-solving is a major part of the engineering design process. Thus there are many 
opportunities for students to demonstrate their problem-solving skills in assessment tasks for this 
area. Such tasks could require them to develop designs, to propose or critique solutions to 
problems after being given criteria and constraints, to select appropriate resources by considering 
trade-offs, to construct and test a model or prototype, to troubleshoot systems and applications, 
or to determine the consequences of making a change in a system. Students must use their 
understandings about the technological principles specified for Design and Systems in chapter 
two as they plan, try out, critique, and revise attempted solutions. 

Communicating and Collaborating 

Communication and collaboration practices are integral to achieving the goals of technological 
design and systems. Students can demonstrate teamwork in tasks where design assignments are 
distributed among team members, progress and results are integrated and shared, and products 
are presented jointly. Designs and the design process can be represented in visual and verbal 
forms. Students can create instructions for system assembly and prepare documentation of a 
procedure for maintaining a system. Students must use their understandings of the technological 
principles specified for both Design and Systems and ICT in chapter two to use tools and 
strategies to communicate and collaborate. Because students are completing the assessment 
individually on the computer, collaborators will be virtual — that is, presented by the computer. 

Table 3.3 offers examples of how the 3 practices can be applied to assessment targets for Design 
and Systems to generate tasks and items for the middle school level. The key principles and 
grade 8 targets were selected from tables 2. 6-2. 9 in chapter two. The chapter two tables present 
simpler versions of the targets for grade 4 and more challenging versions of the targets for grade 
12. Again, these are sample ideas for items and tasks and will not be used in the actual 
assessment. 
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Table 3.3 Examples of grade 8 tasks representing practices 
in each subarea of Design and Systems 



A. Nature of 
Technology 

B. Engineering 
Design 

C. Systems 

Thinking 

D. Maintenance and 
Troubleshooting 

\ Selected 

\ Principles 

Practices \ 

Technologies 
developed for one 
purpose are sometimes 
adapted to serve other 
purposes. 
Technological 
development involves 
creative thinking. 

Requirements for a 
design challenge 
include the criteria 
for success and the 
constraints or limits 
that cannot be 
violated in a solution. 

All technological 
systems have parts 
that work together to 
accomplish a goal. 

Tools and machines 
must undergo regular 
maintenance to ensure 
their proper functioning. 

Understanding 

Technological 

Principles 

Describe the properties 
of a spring that inspired 
the invention of the 
Slinky, (animation) 

List three important 
criteria for a device 
that will toast bread, 
and justify each one. 

How do the physical 
and human 
components of the 
school cafeteria food 
service system work 
together? 

(animation) 

Why do Bill and Sally 
oil their bike chains and 
axles and check the 
brakes each month? 
What may happen if 
they do not? 

Developing 
Solutions and 
Achieving Goals 

Given a collection of 
objects, design a new 
toy (for example, for a 
baby, young child). 
What are the criteria 
for a toy, and how does 
your design meet them? 

Design a process to 
serve 50 slices of 
warm toast in 5 
minutes, given 
specific equipment 
and resources. 

Design modifications 
to the cafeteria for 
people with a 
particular physical 
disability (for 
example, vision, 
mobility). 

(animation) 

Bill’s new bike gears 
are not working 
correctly. What should 
he do? 

Communicating 
and Collaborating 

Select a team of people 
who could design and 
build a new toy for a 5- 
year-old, and justify the 
choices. Work 
individually, or 
collaborate with a 
virtual person to make 
your selections. 

How would an 
industrial toaster 
salesperson develop 
talking points for 
selling a particular 
toaster to a given 
restaurant? 

Collaborate with a 
virtual group of 
people to redesign 
the tasks of the 
cafeteria staff, given 
the new machines. 

Bill, Sally, and many 
other students would 
like to ride their bikes to 
school. Present a design 
for a bike parking lot at 
the school. 
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Illustrative Tasks and Items 


These items from existing tests were selected as examples that could be adapted for assessing 
targets in Design and Systems. Each item is analyzed for the ways in which it would call on the 
practices. 

Scenario-Based Item Set 

In this example, students are asked a series of questions related to a simulation of a nuclear 
reactor. In the NAEP Technology and Engineering Literacy Assessment, the questions might 
relate to all three practices. Students may be asked to demonstrate the practice of Understanding 
Technological Principles by identifying the inputs and outputs of the system or analyzing the 
potential hazards. Students might employ the practice of Developing Solutions and Achieving 
Goals in items asking for results of investigations based on manipulating the simulation to find 
safe levels of temperature and power. The practice of Communicating and Collaborating could 
be elicited in tasks involving virtual (that is, computer-generated) experts in preparing a report of 
findings. 


Context: Energy 
Topic: Nuclear Reactor 
Target Level: Grade 8 


In the nuclear power plant simulation below, the number of nuclear interactions produced by tbe reactor is regulated try the 
Control Rods. The power output is regulated by the Generator Valve. The Control Rods and the Generator Valve can be 
controlled by dragging the sliders up and down. 



set tne generator valve at 80%. Howrar {%) snouid tne control rods be lined tor tne power plant to supply a continuous output or 
800 Megawatts (MW) without the safety valve opening? 

r 55%. 


c 70%. 
r 85% 
r ioo% 


(Source: Organisation for Economic Co-operation and Development, 2005) 
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Conventional Items 

These items will ask for students to select or construct their answers. The examples below are of 
the familiar constructed- and selected-response formats. 

Constructed Response 

Similar to the previous example, this item represents a sample task in which students must use 
their knowledge about the engineering design process to answer a constructed-response item. 
The practice of Understanding Technological Principles is measured by this computer-based 
item. In this item, students use various tools to explore the factors that affect plant growth in a 
greenhouse. In the NAEP Technology and Engineering Literacy Assessment, a similar item 
might ask students to evaluate different greenhouse designs in tenns of their effects on plant 
growth. 
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Context: Plant Growth 
Topic: Designing a Greenhouse 
Target Level: Grade 12 

| Pago 1 1 Paga 2 f Page 3 j Page 4 ] 



Santa and Tom want to 
check how much difference 
the light and warmth can 
make. 

They decide to make one 
end of the greenhouse much 
hotter than the other, and one 
side with much more light 
than the other. 

They put two plants in pots. 
One plant is to be grown as 
fast as possible, the other 
as slowly as possible. 


10°C 20 °C 30 °C 40 °C 




Very 

Dark 

Quite 

Dark 

Quite 

Bright 


Very 

Bright 


Drag each pot to the correct part of the 
greenhouse. 


V7 w7 


The bar chart shows how 
much the plants grew each 
day at different 
temperatures. 

Click on the slider to see 
how the amount of light 
affects the bar chart 



Tom says that the plants 
will grow faster if they keep 
them as hot as possible 



Is Tom right ? Click Yes or No 




Explain your answer in you work book 



Yes 


(Source: Adapted from Ripley, 2009) 



No 
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Selected Response-Multiple Choice 

The following item is another conventional item type — in this case, multiple choice. Students 
would employ the practice of Understanding Technological Principles by demonstrating their 
knowledge about the engineering design process. 


Context: Product Design 
Topic: The Design Process 
Target Level: Grade 12 

An assignment for the design of an emergency light is described in the box below. 


Design and build an emergency light. It must have its own battery, provide a 
bright light, and have a built-in charger. It must be capable of operating while 
being charged. It must have a watertight enclosure with a handle on the top. 


After receiving this assignment, which of the following most likely would be the next step in 
the engineering design process? 

A. testing and evaluation 

B. selecting the best solution 

C. investigation and research 

D. construction of a prototype 

(Source: Massachusetts Department of Elementary and Secondary Education, 2009) 


Practices Applied in Information and Communication Technology (ICT) 

ICT literacy includes the capability to communicate ideas and solutions, to collaborate with peers 
and experts, to conduct research, to investigate solutions to academic and real-world problems, to 
find ways to meet the ever-changing needs of society, to properly acknowledge the source of 
ideas and information, and to select and use appropriate digital tools. The sections below 
describe how the three practices apply to the ICT assessment targets. 

Understanding Technological Principles 

The principles in the ICT assessment area involve understanding the variety of ICT tools and 
how and when they can be used to accomplish a wide variety of tasks in school and in practical, 
out-of-school situations. Students will need to know about the general features and functions of 
types of ICT tools, which tools are appropriate for particular purposes, and to understand criteria 
for detennining if the tools were used appropriately and well. 


3-17 


Technology and Engineering Literacy Framework for the 2014 NAEP 


Developing Solutions and Achieving Goals 

ICT tools can be employed to support problem-solving and achievement of goals in all three of 
the technology and engineering literacy areas. The types of problems addressed in the ICT 
assessment area relate to the selection and use of appropriate tools to achieve goals related to 
information research, investigating problems, meeting the needs of society, constructing and 
exchanging information and ideas, and acknowledging ideas and infonnation. ICT problem- 
solving practices could be elicited by tasks and items asking students to select and use 
applications effectively and productively; to access and use infonnation and data to solve a 
problem or achieve a goal; to use ICT tools to solve a problem or achieve a goal; or to use ICT 
tools to plan an approach to solving a problem, represent data, analyze results, and summarize 
and present findings. Students must use their understandings about the features and functions of 
ICT tools in order to apply the tools appropriately and effectively to develop solutions and 
achieve goals for given tasks and problems. 

Communicating and Collaborating 

ICT capabilities rely heavily on students’ command of communication and collaboration skills. 
Students can be asked to demonstrate the capability to contribute effectively to a body of 
knowledge or to take part in group deliberations through social media (simulated in the 
assessment) and the use of other contemporary communication tools and structures. Students can 
be asked to investigate a problem or pursue a goal as individuals or with a group (virtually), to 
integrate input (provided in the task) from multiple collaborators (virtual) who are peers or 
experts, and to reach consensus. Students can integrate feedback from others, provide 
constructive criticism, and communicate to multiple audiences using a variety of media and 
genres. Findings can be represented in a variety of ways, such as diagrams, tables, graphs, and 
digital media. To communicate and collaborate effectively, students will need to use their 
understandings about features and functions of communication and collaboration tools and apply 
their knowledge and understanding of the ICT tools and strategies. 

Table 3.4 provides some examples of how the 3 practices can be applied to assessment targets 
for Infonnation and Communication Technologies to generate tasks and items at the middle 
school level. The key principles and grade 8 targets were selected from tables 2.10-2.14 in 
chapter two. Those tables further describe simpler targets at grade 4 or more complex targets at 
grade 12. These are sample ideas for items and tasks and will not be used in the actual 
assessment. 
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Table 3.4 Examples of grade 8 tasks representing practices that apply to each subarea of 
Information and Communication Technologies (ICT) 



A. Construction 
and Exchange of 
Ideas and 
Solutions 

B. Information 
Research 

C. Investigation 
of Academic 
and Real-World 
Problems 

D. Acknowledg- 
ment of Ideas 
and 

Information 

E. Selection 
and Use of 
Digital Tools 

\ Selected 

\ Principles 

Practices \ 

Digital tools offer 
many options for 
formal and 
nonformal 
expression in 
nearly every 
academic and 
professional 
discipline. 

Important 
strategies for 
ensuring quality 
of information 
include 1) 
assessing the 
source of 
information and 
2) using multiple 
sources to verify 
the information 
in question. 
Information can 
be distorted, 
exaggerated, or 
otherwise 
misrepresented. 

Digital tools can 
be very helpful 
in generating 
ideas and solving 
problems in 
academic 
subjects as well 
as in researching 
practical 
problems. 

There are 
multiple 
guiding 
principles 
(laws, policies, 
and guidelines) 
that govern the 
use of ideas and 
information. 

Knowledge 
about the 
common uses 
of readily 
available 
digital tools 
supports 
effective tool 
selection. 

Understanding 

Technological 

Principles 

Describe how 
graphics, text, and 
tables convey a 
message. 

From among a 
group of Web 
pages select the 
particular one 
that contains an 
example of 
exaggeration. 

Identify the 
combination of 
data tools that 
will best present 
a particular set of 
data. 

Select the 
example that 
shows an 
appropriate way 
to give credit to 
another 
student’s 
graphic. 

Critique a 
digital tool 
suggested for 
designing an 
online story. 

Developing 
Solutions and 
Achieving Goals 

Develop an online 
survey for 
elementary school 
students 
concerning the 
design of a new 
playground. 

Create a digital 
story about a 
historical period 
by choosing 
images of art 
from the period. 

Use simulations 
and 

visualizations to 
describe the rate 
of deforestation 
in Brazil. 

Identify which 
online images 
can legally be 
used in a 
student 
presentation. 

Predict trends 
in rates of 
software 
piracy based 
on provided 
data. 

Communicating 

and 

Collaborating 

Respond to 
suggestions from 
two virtual 
(computer- 
generated) 
collaborators 
explaining why the 
search results of 
only one of the 
collaborators have 
sufficient 

information for the 
report. 

Ask a virtual 
collaborator for 
help on 
developing a 
digital 

presentation. 

Enter costs from 
several sources 
and 

communicate to 
the principal the 
most economical 
printer for school 
play posters. 

Post a 

copyright-free 
image to a 
website and 
communicate to 
friends that it is 
available. 

Use two 
digital tools to 
create a public 
service 

announcement 
on software 
piracy. 
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Illustrative Tasks and Items 


The following tasks and items illustrate how the three practices apply to ICT assessment targets. 
The practices may be elicited by extended, scenario-based task and item sets or by separate 
items. 

Scenario-Based Item Sets 

In this simulation, students navigate among a file manager, an email client, a Web browser, a 
word processor, and a spreadsheet to make a travel brochure for a fictional town, Pepford. They 
are assessed on how they use these ICT tools to accomplish the task, not on the quality of the 
brochure. The process is more important than the outcome. Students use the practice of 
Understanding Technological Principles by calling on their knowledge of which ICT tools to use 
to accomplish the goal and how to use them. Students also apply the practice of Developing 
Solutions and Achieving Goals as they conduct their information research. The practice of 
Communicating and Collaborating is not assessed directly in this task, although it could be added 
by including tasks assessing the quality of the brochure for communicating to the intended 
audience and tasks involving the assignment of information gathering from virtual (computer- 
generated) collaborators and the integration of their (virtual) input into the brochure. 


Context: Travel and Tourism 
Topic: Promotional Brochure 
Target Level: Grade 8 



(Source: Ripley, 2009) 
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Conventional Items 


These items will ask for students to select or construct their answers. The examples below are of 
the familiar constructed- and selected-response formats. 

Constructed and Selected Response 


In this task, students are presented with a list of websites from the results of a Web search. 
Students must decide which website has the most reliable information and justify their response. 
The practice of Understanding Technological Principles is applied to students’ knowledge of 
criteria for how to judge the credibility and quality of information sources — one of the ICT 
targets in Information Research. 


Context: Medicine 
Topic: Web Search 
Target Level: Grade 8 


DIRECTIONS: 

You have searched the Internet for general 
information about treating bee stings and found the 
sites shown. 

Pick one site that you think has reliable information 
about treating bee stings Type the letter of that site 
(A-J)inthe box below. 


□ 


Explain why you chose this site. 
Type your answer below 




SJh) 


Advance foot? Preterenre ? lan guage Jq.qIi 
|treat bee stings [ Search ] 


Bee Venom Therapy (Bee Stings! I AHealthyMe.com 
A. Find out whether subjecting yourself to bee stings can help ease the pain., 
bee venom to successfully treat rheumatoid arthritis 

www.ahealthyme com/topic/beevenom - 49k - Cached - Similar pages 


F gm ih g ? fp . r Na tural Living 

B. Have treatment brought to them When I was a child, my mother always 
bee stings and other venomous. . . 

famihesfornaturalliving orq/herbs/summerrem.htm - 16k - Cached - Similar pages 


Great for Bee Stings 

C. Nova Skin Rash Complex is Great for Poison Ivy, Poison Oak, Hives, Bee 

Stings, and Insect Bites. ... diagnosing or treating a health problem or disease 

www.painpharm com/mall/n_prod_skinrash asp - 41k - Cached - Similar pages 


MEDLINEpIus Medical Encyclopedia: Insect bites or stings 

D. develop severe allergic reactions (anaphylaxis) to insect stings the sting 
is from a honey bee, remove the stinger treat the victim 

www.nlm.nih gov/medlineplus/ency/article/D00033 htm - 43k - Cached - Similar 
pages 


Treating Bee Stings - DrGreene.com - caring for the next ... 

B. was swollen. Is this serious? How do you treat bee stings? What insect 
repellents work the best at preventing them? California. Bee 

www.drgreene.com/21 202.html - 49k - Cached - Similar pages 



(Source: Lennon et al., 2003) 
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Selected Response — Multiple Choice 

This multiple-choice item also assesses a student’s capability to evaluate websites, an example of 
the practice, Understanding Technological Principles. This example also demonstrates a 
computer-based selected-response format. 


Context: The Internet 
Topic: Website Search 
Target Level: Grade 12 



(Source: Australian MCEECDYA, 2008) 
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Contexts 

Technology and engineering literacy requires not just that students know about technology but 
also that they are able to recognize the technologies around them, understand the complex 
relationship between technology and its effects on society, and use technological principles and 
tools to develop solutions to problems and meet goals. Consequently, NAEP Technology and 
Engineering Literacy Assessment items will measure students’ technology and engineering 
literacy in the context of relevant societal issues, design problems, and school and community 
goals. As the three areas of technological literacy to be measured by NAEP tend to focus on 
somewhat different types of issues, problems, and goals, the contexts and situations that will 
frame the technology assessment items in these areas will differ somewhat as well. 

Contexts in Technology and Society 

The complex and multifaceted interactions between technology and society often manifest 
themselves in unexpected and unpredictable ways as new technologies are used in particular 
contexts or situations. A new technology may succeed in meeting the need that it was intended to 
meet and bring about far-reaching benefits, but it may also have negative, unintended 
consequences. For example, mobile communication devices have transformed business and 
personal interactions, yet a large number of traffic accidents have been blamed on drivers using 
these devices while operating their vehicles. Similarly, farming practices have increased crop 
production, but they have also risked contaminating sources of groundwater. Such issues can 
clearly be used as contexts for NAEP Technology and Engineering Literacy Assessment items 
(as illustrated in ISTE’s NETS*S, the framework of the Partnership for 21st Century Skills, and 
ITEEA’s Standards for Technology Literacy). The contexts for tasks and items in the area of 
Technology and Society may include such technologies with positive and negative effects and 
may also present ways that technology improves people’s lives, such as water purification, 
sewage treatment, and medicine, or the various ways that people regularly interact with 
technology, from brushing teeth in the morning to crawling into a warm, comfortable bed at 
night. The following are examples of topics in the contexts of health, energy, and electronic 
communications that could be used to generate assessment tasks and items for sample targets in 
the 3 subareas of Technology and Society for grade 8: 

• Agriculture and health contexts: water as a scarce resource; 

• Energy context: wind turbines for homes; and 

• Electronic communications context: personal communication devices. 
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Table 3.5 Examples of how different contexts may be used to generate 
tasks and items for Technology and Society for grade 8 



A. Interaction of 
Technology and 
Humans 

B. Effects of 
Technology on the 
Natural World 

C. Effects of 
Technology on the 
World of 
Information and 
Knowledge 

D. Ethics, Equity, 
and Responsibility 

Key Principles 

Society drives 
technological change, 
while changing 
technologies in turn 
shape society. 

Some technological 
decisions put 
environmental and 
economic concerns in 
competition with one 
another, while others 
have positive effects 
for both the economy 
and the environment. 

Information 
technology is 
evolving rapidly, 
enabling ever- 
increasing amounts 
of information and 
data to be stored, 
managed, enhanced, 
analyzed, and 
accessed through a 
wide array of devices 
in various media 
formats. 

Technology by itself 
is neither good nor 
bad, but its use may 
affect others. 

Context: 
Agriculture and 
Health: Water 
as a Resource 

What societal needs 
drove the changes 
made to a river’s 
natural flow? 

What issues need to 
be addressed to 
ensure that the water 
system stays healthy? 

Find two reports 
describing alternative 
water purification 
methods. 

How might the 
decision to divert 
water from the rivers 
affect farmers and 
small towns 
downstream? 

Context: 
Energy: Wind 
Turbine 

Describe the positive 
and negative impacts 
that residential wind 
turbines might have 
on society. 

Compare and contrast 
the environmental 
and economic 
impacts of wind 
turbines with other 
potential sources of 
energy. 

Compare the 
persuasiveness of two 
multimedia 
presentations on 
alternative wind 
turbine designs. 

Describe a process 
for citizens to 
evaluate the effect 
that wind turbines 
might have on others 
in the community. 

Context: 

Electronic 

Communication 

What are the positive 
and negative effects 
that personal 
communication 
devices may have on 
traditional human 
communication? 

Describe the effect of 
video evidence of 
environmental 
destruction on 
society’s awareness 
of the global impact 
of pollution. 

Describe ways that 
personal 
communication 
devices provide 
access to information 
and expertise. 

What might the effect 
be of allowing 
personal 
communication 
devices to be used in 
school? 
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Contexts in Design and Systems 

Nearly all of the products and processes in human society result from the development of one or 
more kinds of technology. Homes, factories, and farmhouses are built using construction 
technologies. Fruits and vegetables are grown and processed using agricultural technologies and 
are brought to market and to the dinner table with transportation technologies. Methods of 
extracting and using fuels to produce power involve energy and power technologies, and the 
tools and processes used by doctors, nurses, and phannacists are a part of medical technologies. 
Although these technologies can be classified in various ways, in order to provide guidance to 
item writers, this framework identifies the following technology areas that can be used as 
contexts to measure students’ understanding of design and systems (drawn primarily from 
ITEEA, 2007): 

• Agricultural and related biotechnologies; 

• Construction technologies; 

• Energy and power technologies; 

• Infonnation and communication technologies; 

• Materials and manufacturing; 

• Medical technologies; and 

• Transportation technologies. 

The section on the next page presents potential scenario topics placed in contexts from the types 
of technologies listed above. The table illustrates how the topics in these contexts can be used to 
generate tasks and items in the 4 subareas of Design and Systems for grade 8. The scenarios 
would be simpler for grade 4 and more complex for grade 12. 
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Table 3.6 Examples of how different contexts may be used to generate 
tasks and items for Design and Systems for grade 8 



A. Nature of 
Technology 

B. Engineering 
Design 

C. Systems 

Thinking 

D. Maintenance 
and 

Troubleshooting 

Key Principles 

Scientists are 
concerned with what 
exists in nature; 
engineers modify 
natural materials to 
meet human needs 
and wants. 

Requirements for a 
design challenge 
include the criteria 
for success, or goals 
to be achieved, and 
the constraints or 
limits that cannot be 
violated in a 
solution. 

Systems may include 
subsystems and may 
interact with other 
systems. Systems 
may also be 
embedded within 
larger systems. 

Tools and machines 
must undergo 
regular maintenance 
to ensure their 
proper functioning. 

Context: 

Transportation 

How have 
transportation 
methods changed 
over time? 

Propose two ways to 
modify an 
intersection to make 
it safer. 

What are the 
advantages of 
container cargo ships 
over other ways to 
transport goods to 
market? 

What problems 
might occur if 
engines are not oiled 
periodically? 

Context: 

Medical 

Technology 

What were the 
technological 
advances that 
allowed medical 
researchers to 
develop vaccines? 

Identify the 
requirements for a 
prosthetic arm that 
will enable a person 
to play tennis. 

Name several 
elements of the 
nation’s medical 
system and describe 
how they are related. 

What procedures 
would you 
recommend for 
maintaining the 
school’s first-aid 
kits? 

Context: 
Energy: Wind 
Turbine 

What natural 
constraints exist in a 
city that might cause 
a homeowner to 
choose wind power 
over other “green” 
energy alternatives? 

Compare the 
aesthetic qualities of 
the two types of 
wind turbines 
(vertical or 
horizontal). 

Using the simulation 
model of a 
residential wind 
turbine, identify the 
goals, inputs, 
processes, outputs, 
and feedback and 
control features. 

Using the simulation 
model of a 
residential wind 
turbine, describe 
which parts of the 
machine would 
require the most 
maintenance. 

Context: 
Information and 
Communication 
Technologies 

Trace the evolution 
of features on early 
cellphones compared 
with current 
smartphones. 

Compare the trade- 
offs of functions 
available in two 
specific devices 
designed for a 
workplace or 
personal use. 

Explain two ways in 
which personal 
communication 
devices can work 
together for a team 
to achieve its project 
goal. 

Describe a set of 
troubleshooting 
steps that would be 
appropriate for 
analyzing a problem 
with a printer. 
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It is important to note that students are not expected to be familiar with the specific components 
and working details of any particular technology. For example, they will not be tested on their 
knowledge of genetic engineering, an important biotechnology, or on their understanding of 
energy and power or networking technologies. While these topics may be used to provide the 
context for test items, the information required for students to respond to the test questions will 
be provided in the scenario or background of the question. Students will be tested on the broad 
set of principles concerning design and systems and capabilities described in chapter two. 
However, one of the technologies from the list in the previous section has been chosen for more 
emphasis in the 2014 NAEP Technology and Engineering Literacy Framework, and that is 
Infonnation and Communication Technology. 

Contexts in Information and Communication Technology (ICT) 

In contrast to other types of technologies, students will be expected to be fluent in the use of 
information and communication technologies, as described in the tables in chapter two. The 
reason for this additional attention to ICT is that it is pervasive in society, and some level of 
technology and engineering literacy is required for virtually every profession, in every school 
subject, and in all walks of life. Furthermore, it is likely that literacy with information and 
communication technologies will become even more important in the decades ahead. 

Because of the ubiquity of ICT, it is difficult to describe all of the particular contexts for items 
that NAEP will design to assess students’ knowledge of it and their capabilities to put it to use. 
ICT knowledge and skills can be applied in the context of developing and using any of the 
technologies included in table 3.7, and it can be applied to any of the ways that technology 
interacts with society. ICT principles and tools should be a part of every person’s set of 
capabilities in and out of school for solving problems or working to meet a goal. ICT tools have 
become integral supports for learning school subjects. People who are literate in technology and 
engineering should be able to select and use technological tools to research a period in history, 
compare cultures, collect and display data in a scientific investigation, develop a story or 
presentation, or produce a work of art. The types of scenarios used to assess students’ knowledge 
and skills in this area will require that the item provide an opportunity for students to 
demonstrate their understanding of and capabilities to use ICT to address goals and problems in 
Technology and Society, in Design and Systems, and, more generally, in various disciplines, and 
in real-world, practical applications. The following table illustrates how topics set in different 
contexts can be used to generate tasks and items for targets in the five subareas of ICT. 
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Table 3.7 Examples of how different contexts may be used to generate tasks and items for 
Information and Communication Technology (ICT) for grade 8 



A. Construction 
and Exchange 
of Ideas and 
Solutions 

B. Information 
Research 

C. Investigation 
of Problems 

D. Acknowledg- 
ment of Ideas 
and Information 

E. Selection 
and Use of 
Digital Tools 

Key 

Principles 

Digital tools offer 
many options for 
formal and 
nonformal 
expression in 
nearly every 
academic and 
professional 
discipline. 

Important 
strategies for 
ensuring quality of 
information 
include 1) 
assessing the 
source of 
information and 2) 
using multiple 
sources to verify 
the information in 
question. 

Digital tools can 
he very helpful in 
generating ideas 
and solving 
problems in 
academic subjects 
as well as in 
researching 
practical 
problems. 

Fair use guidelines 
are designed to 
support the use of 
copyrighted 
materials for 
academic purposes 
and for journalism 
and other forms of 
writing and 
commentary. 

A fundamental 
aspect of 
technological 
literacy is the 
possession of 
foundational 
ICT skills in the 
use of common 
productivity 
tools. 

Context: 

Practical 

Applications: 

Local 

Community 

Action 

Web pages have 
been created to 
persuade people to 
move to your 
community. Use a 
rubric to evaluate 
mock-up samples 
of Web pages, 
including the 
appropriate use of 
media to provide 
information to 
appropriate 
audiences and also 
the appropriate 
citation of sources. 

Synthesize data 
from a variety of 
sources (census, 
local economy, 
demographics, 
industry, history) 
to show the 
availability of jobs 
in your town. 

Watch an example 
video of a 
successful local 
campaign to 
declare a local 
building as a 
historic landmark. 
Answer a series of 
questions about 
the effectiveness 
of the video. 

Review examples of 
media that could be 
used for a 
presentation at a 
local community 
meeting and then 
make decisions as to 
the appropriateness 
and legality of using 
them. 

Using articles 
and simulated 
websites, create 
a media-rich 
presentation 
designed to 
persuade people 
to move to your 
city. 

Context: 

Energy: 

Wind 

Turbine 

Using information 
from the text and 
the provided slide- 
making tools, 
design a 

presentation that 
presents the 
positive and 
negative impacts 
of residential wind 
turbines to a group 
of homeowners. 

Using the 
provided Internet 
search and slide- 
making tools, 
research and 
create a 
presentation 
comparing and 
contrasting the 
benefits of 
residential wind 
turbines with other 
“green” energy 
alternatives. 

Using the 
simulation model 
and the available 
city data, 
determine which 
cities in the United 
States would be 
most appropriate 
for installing 
residential wind 
turbines. Be sure 
to use data from 
the simulation to 
support your 
conclusions. 

Examine a set of 
slides and associated 
resources from a 
wind turbine 
manufacturer and 
determine how best 
to give appropriate 
credit for the 
information and 
images used in a 
presentation to 
promote residential 
wind turbines. 

Using the 
information 
provided in the 
text, choose 
from the 
available digital 
tools and create 
a multimedia 
information 
packet to 
promote 
residential wind 
turbines. 

Context: 

School 

Subject: 

History 

Taking the role of 
a student during 
the Civil War, 
refer to examples 
from the online 
archives to 
compose a letter 
and photo essay 
describing the 
First Battle of Bull 
Run. 

Conduct research 
in online libraries 
(for example, 
Library of 
Congress) 
synthesizing news 
accounts and 
pictures on the 
events and their 
impacts. 

Working with 
online archives, 
compare the 
hardship endured 
by civilians during 
the First Battle of 
Bull Run and the 
Battle of 
Gettysburg. 

Develop credits and 
acknowledgments 
for your online 
sources. 

Explain how the 
ICT tools you 
used contributed 
to a presentation 
on the Battle of 
Gettysburg. 
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The intent of placing technology and engineering literacy assessment tasks and items in a range 
of contexts is to ensure that students have the opportunity to demonstrate what they know and 
can do across many types of problems and situations. By sampling student knowledge and skills 
in a variety of contexts and practices, the assessment will provide a broad and deep picture of 
technology and engineering literacy proficiencies. 
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Chapter Four: Overview of the Assessment Design 


Introduction 

This chapter provides an overview of the major components of the assessment design. It begins 
with a brief description of the 2014 NAEP Technology and Engineering Literacy Assessment 
and follows that with a discussion of the types of assessment tasks and items, how they can be 
used to measure student understanding and skills, how students will respond to these tasks, and 
how their responses will be monitored and evaluated. In addition, this chapter describes how the 
assessment should be balanced across the major assessment areas in technology and engineering 
literacy as well as across the practices. The types of items to be included in the assessment are 
described, and examples are provided. English language learners and students with disabilities 
were recognized from the start as an important part of the student population to be included in 
the 2014 NAEP Technology and Engineering Literacy Assessment, and thus were considered as 
the assessment framework was developed. Concepts of universal design for assessment were 
considered as one way for the assessment to be more inclusive of all students. 

Overview of the NAEP Technology and Engineering Literacy Assessment 

In 2014, the NAEP Technology and Engineering Literacy Assessment will be conducted at the 
national level for grade 8. The assessment will include tasks and items sampled from the domain 
of technology and engineering literacy achievement identified by the intersection of the three 
major areas of technology and engineering literacy and the crosscutting practices. The 
assessment will be administered by computer and will be composed of sets of long scenarios, 
short scenarios, and discrete items. Within each of these types of tasks there will be a variety of 
selected-response items and short and extended constructed-response items. Student responses 
will be measured both directly and, in the scenario-based tasks, through their interactions with 
simulated tools and their manipulation of components of systems. 

Types of Tasks and Items 

Allowing students to demonstrate the wide range of knowledge and skills detailed in the NAEP 
Technology and Engineering Literacy Assessment targets will require a departure from the 
typical assessment designs used in other NAEP content areas. Thus students will be asked to 
perform a variety of actions using a diverse set of tools in the process of solving problems and 
meeting goals within rich, complex scenarios that reflect realistic situations. Consequently, this 
assessment will rely primarily on scenario-based assessment sets that test students through their 
interaction with multimedia tasks that include conventional item types, such as selected-response 
items, and also monitor student actions as they manipulate components of the systems and 
models that are presented as part of the task. 

The following sections describe in detail the scenario-based assessment sets and the sets of 
discrete, conventional items that will be developed for the NAEP Technology and Engineering 
Literacy Assessment. Note that the examples of items shown are drawn from what is available at 
the time of developing this framework (spring 2010). It is expected that, as the use of such 
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innovative items expands and technology advances, these examples will become dated, but they 
still illustrate the principle components that are important in this assessment. 

The assessment will be administered to a nationally representative sample of students to report 
on student achievement at the group level. The assessment is not designed to measure the 
performance of any individual student or school. To obtain reliable estimates across the 
population that is tested, a large pool of assessment items will be developed. That pool of items 
will be too large to give to any individual student, so subsets of items will be selected to 
administer to each individual student. The NAEP Technology and Engineering Literacy 
Assessment will be given in approximately 50 minutes, with additional time for background 
questionnaire completion. The assessment sets that will be developed for the NAEP Technology 
and Engineering Literacy Assessment are described below. 

Scenario-Based Assessment Sets 

There will be two types of scenario-based assessment sets, one long and one short. The long 
scenarios will take students approximately 25 minutes. The short scenarios will take students 
about 12 to 15 minutes to respond. The two types of scenarios have common characteristics, but 
they differ in complexity and in the number of embedded assessment tasks and items to which a 
student is asked to respond. Long scenarios will provide about 10 to 15 measures of 
performance, and the short scenarios will capture approximately 5 to 10 measures of 
performance. Measures will include innovative measures of a student’s interaction with aspects 
of the scenario as well as conventional selected-response items and short constructed-response 
items. The different measures are discussed later in this chapter. 

Discrete Item Sets 

One of the challenges for this assessment is that the use of scenario-based assessment sets 
reduces the number of independent measures in the assessment as a whole. Because of their 
capability to replicate authentic situations examinees may encounter in their lives, scenarios have 
the potential to provide a level of authenticity other types of assessment tasks cannot provide, 
which, in turn, may contribute to the validity of the entire assessment. At the same time, 
however, the choice to use these complex tasks reduces the number of measures that can be 
included in any one test and causes many of the measures to be interdependent because they are 
related to the same scenario. To counteract this interdependency and ensure reliability, the NAEP 
assessment of technology and engineering literacy will also include sets of discrete items that 
produce independent measures. Discrete item sets will include conventional selected-response 
items and short constructed-response items. 

Definitions of the Scenario-Based and Discrete Item Assessment Sets 

Scenario-Based Assessment Sets 

As this framework is written, the use of computer-based scenarios for assessment purposes is an 
emerging but growing area. The 2009 NAEP Science Framework called for the use of interactive 
computer tasks as part of its assessment, and in 2009, 3 long and 6 short interactive tasks were 
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administered to national samples of students in grades 4, 8 and 12. Another set of interactive 
assessment tasks has been used in state tests of science achievement in an Enhanced Assessment 
Grants project funded by the U.S. Department of Education to determine how simulation-based 
scenarios in science might form part of district and state accountability systems. As assessment 
developers gain more experience in this emerging field, they will develop a better sense of how 
to create the tasks efficiently and how to ensure that the tests produce valid and reliable 
measures. 

Here a descriptive outline is provided of the main features of the scenarios that will be developed 
for the NAEP Technology and Engineering Literacy Assessment. At the beginning of the 
scenario, it is important to set the context for the activities in which the student will be involved. 
This introduction provides a setting for the assessment tasks that, as far as possible, should 
reflect tasks that might be performed in society, either within an academic setting or outside of 
school. In addition, near the start of the scenario, a motivating question or goal will be 
introduced. This goal provides the driving rationale for the tasks that the student will perfonn, 
and it offers a storyline that helps define the relevance and coherence of the tasks and motivates 
the student to undertake them. The motivating goal might be to solve a particular problem or to 
achieve a certain goal within the scenario. 

An advantage of computer delivery of the assessment is that the introductions to the scenarios 
can use appropriate multimedia to present the settings for the assessment tasks. As a result, there 
is less need for text and therefore less of a reading demand. The multimedia can include video 
segments or animations that the student observes, and it will also generally use text, numbers, 
and graphics to convey information necessary for the tasks to be accomplished. All of the 
representations, such as graphics, video, or simulations, must be carefully chosen to serve a 
purpose in the assessment tasks, and none should be present simply for visual interest. 

One type of assessment scenario will include a representation of a system. Depending on the 
context for the particular scenario, this might be an engineering system such as an irrigation 
system or a dam. Whatever the system, it will have components that are dynamically related, so 
that a student can observe the role of a particular component (for example, watch what happens 
when a valve is opened in an irrigation system) or interact with a component (for example, by 
setting a value for a parameter or moving a part of the system) and see a resulting change of state 
in the system (for example, a rise in water levels or flooding of fields). 

A second type of scenario will lay out an overarching goal or problem that students will reach or 
solve by conducting various interrelated tasks. Such a goal might be, for instance, to develop a 
play about a historical period. Component tasks could involve searching for infonnation about a 
famous incident in the period, analyzing the necessary ticket sales, and creating a playbill 
advertising the production. 

Within a scenario, students may be asked to select tools from a toolkit and use them within the 
system. Students might be asked, for example, to select a graphing or spreadsheet tool or to use a 
simulation. Various tools may be made available, depending on the scenario. Word-processing, 
texting, or presentation tools might be available for communication tasks, for example, and Web 
design or page layout tools might be used for the presentation of large amounts of information. 
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For some scenarios it might be appropriate to provide more specialized tools, such as computer- 
aided design, geographical information systems, or video editing tools. 

By interacting with the components of the system or task that are key elements for achieving the 
goal, students are able to respond to tasks that ask them to explore alternative outcomes, control 
certain variables, and observe the resulting changes in the system. The students can observe and 
describe the patterns or characteristics of the outcomes and can interpret the feedback from the 
system. They can then evaluate the outcomes of the choices they made in manipulating the 
components of the system or in using particular tools, and, finally, they can form conclusions. 

In some cases it might be necessary to simulate virtual features of real-world equipment that can 
be used within a scenario. For example, a temperature gauge might give feedback from a heating 
system, an anemometer might be used to measure wind speed in a scenario about wind turbines, 
or a table might be used in an ICT scenario requiring the collection of data about the types of 
symbolism in Shakespeare’s plays. Alternatively, graphics or images might be constructed or 
selected to communicate a design or idea. 

In providing tools in a scenario, it is necessary to detennine which elements of a tool are 
necessary for the activities in the scenario and which features of the tool will be used by students. 
It is not necessary to provide or simulate a fully featured version of a tool. For example, only 
certain functions of a spreadsheet tool might be provided in order that students could take a table 
of data resulting from actions in the system and transfonn it into a graphical representation of 
their choice (a line graph, say, or a bar graph or pie chart). It would not be necessary to provide 
all of the other features of the spreadsheet tool, and, in fact, it would be distracting to students 
and produce measurement “noise.” 

Throughout their interaction with a system, students may be asked to use tools to find relevant 
resources; to communicate to others about their actions, decisions, or results (for example, 
texting a virtual team member); or, at the end, to convey their conclusions (for example, creating 
a slideshow presentation). 

Discrete Item Sets 

The discrete item sets will comprise approximately 10 to 15 stand-alone items in either selected- 
or constructed-response format to be completed within a 25 -minute block. These items would not 
be part of a complex scenario or related to one another. Each discrete item would provide a 
stimulus that presents enough infonnation to answer the particular question posed in the stem of 
the item. Items in discrete sets will be selected-response items (for example, multiple choice) or 
short constructed-response items in which a student writes a text-based response. 

Descriptions of the Response Types Used in the Assessment Sets 

In conventional items on previous NAEP assessments, students have responded either by 
selecting the correct response from among a number of choices or else by writing a short or long 
text-based response to the questions posed. In the computer-based NAEP Technology and 
Engineering Literacy Assessment, with its scenario-based assessment sets, there are 
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opportunities to greatly extend the ways in which a student can respond to an assessment task. 
Thus this assessment can move beyond the old ways of thinking of response types in terms of 
simply multiple-choice or written responses and begin to consider new types of responses. 

In this assessment, three response types are used: short constructed response, long constructed 
response, and selected response. Although these are the same names as used in other NAEP 
assessment frameworks, in the context of the NAEP Technology and Engineering Literacy 
Assessment they have different and expanded meanings. These meanings are described in the 
following sections. 

Constructed Response 

Constructed responses are ones in which the student “constructs” the response rather than 
choosing a response from a limited choice of alternatives, as is the case with selected-response 
items. Constructed responses in the NAEP Technology and Engineering Literacy Assessment 
will include short constructed-response tasks and items as well as extended constructed-response 
tasks and items. These are described in detail in the following sections. 

Short Constructed Response 

Short constructed responses might be used in either the discrete-item assessment sets or in the 
scenario-based assessment sets. They generally require students to do such things as supply the 
correct word, phrase, or quantitative relationship in response to the question given in the item, to 
identify components or draw an arrow showing causal relationships, to illustrate with a brief 
example, or to write a concise explanation for a given situation or result. Thus students must 
generate the relevant information rather than simply recognize the correct answer from a set of 
given choices, as is the case in selected-response items. When used as part of a discrete item set, 
all of the background information needed to respond is contained within the stimulus material. 

The following is an example of a short constructed-response item that might be used in a discrete 
item set. In this computer-based item, students use a spreadsheet program to create a pie chart. 
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Extended Constructed Response 

Extended constructed responses will be used in the long scenario-based assessment sets. In a 
scenario-based assessment set, the real-world scenario is developed and elaborated on as the 
student moves through the assessment set. As previously described, the introduction of the 
scenario will provide context and motivation for the tasks in the assessment set. As the scenario 
builds, the student undertakes a series of tasks that combine to create the response. For example, 
a student might be asked to enter a search term to gather infonnation about a famous composer 
and to request information from virtual team members. Students could vary the size of 
populations to test a model of a city’s transportation system, or, in a different scenario, they 
might be asked to construct a wind turbine from a set of virtual components in which there are 
several combinations of turbine blades and generators. 

Additional measures of the students’ responses can be made by capturing data about which 
combinations of components the students selected, whether they covered all possible 
combinations, and what data they chose to record from their tests of the components. A follow- 
on task might ask the students to select different types of graphic representations for the 
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tabulated data they captured. Observing whether they have selected an appropriate type of graph 
provides additional information about how they use data analysis tools. 

Finally, the students could be asked to interpret their data, make a recommendation for the best 
combination of turbine blade and generator, and justify their choice in a short written (typed) 
response. In this way, both the task and the response are extended. 

Thus, unlike short constructed-response items in which all the infonnation to answer a particular 
task is contained in a single stimulus, the information necessary to answer an extended 
constructed response is contained in several parts of the overall task. In this example, it would 
not be possible for a student to make recommendations about which combination of blades and 
generators is best without having done all parts of the previous tasks. 

Designing extended constructed-response tasks presents certain challenges. Enough information 
must be provided in the scenario to allow the student to perfonn well-defined, meaningful tasks 
that yield measurable evidence about whether the student possesses the knowledge and skills 
defined in the assessment targets. Another challenge is to ensure that the dependencies among 
the tasks that a student performs within an extended response are minimized. For example, in the 
wind turbine scenario described above, students could run tests of combinations of certain 
turbine blades and generators, and their responses could be assessed. Then, the students could be 
given data from another set of tests of different blades and generators that someone else did and 
asked to interpret those data. In this way the dependency between a student’s own data-gathering 
and the data analysis stage is minimized. The goal is to make sure that mistakes or deficiencies in 
the first part of the task are not carried forward into the second task, thereby giving all students 
the same opportunity to show their data analysis skills, regardless of how well they did on the 
first task. 

In the following simulation, students are given the scenario of a population of small birds — 
chortlers — whose population is declining. The students are asked to use various tools to analyze 
data in a variety of ways and to detennine some possible causes for the population decrease so 
that they can then present their findings on the impacts on the chortlers. The multiple 
developments of the graphs are extended-constructed responses, as is the development of the 
conclusions to be presented, which in this example are not written on the computer. 
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(Source: Ripley, 2009) 


Extended responses can provide particularly useful insight into a student’s level of conceptual 
understanding and reasoning. They can also be used to probe a student’s capability to analyze a 
situation and choose and carry out a plan to address that situation, as well as to interpret the 
student’s response. Students may also be given an opportunity to explain their responses, their 
reasoning processes, or their approaches to the problem situation. They can also be asked to 
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communicate about the outcomes of their approach to the situation. Care must be taken, 
however, particularly with fourth graders and English language learners, that language capability 
is not confounded with technology and engineering literacy. 

Selected Response 

As the name implies, selected-response items are those in which students read a question and are 
presented with a set of responses from which they choose the best answer. In other NAEP 
assessments, selected-response items most often take the form of multiple-choice items, in which 
students select an answer from, say, four options provided. The choices include the most 
applicable response — the “answer” — as well as three “distractors.” The distractors should appear 
plausible to students but should not be justifiable as a correct response, and, when feasible, the 
distractors should also be designed to reflect current understanding about students’ mental 
models in the content area. The NAEP Technology and Engineering Literacy Assessment will 
include such multiple-choice items within both the scenario-based and the discrete-item 
assessment sets as one type of selected response. 

In addition to the conventional multiple-choice selected-response items, the scenario-based sets 
in the NAEP Technology and Engineering Literacy Assessment will include other types of 
selected-response formats. The computer-based nature of the scenarios will allow a variety of 
types of student selections to be measured. For example, a student might be given a task to 
perform and asked to select an individual tool from a set of virtual tools. When a student selects 
a tool by clicking on it, it provides a measurable response that is, in essence, a selected response. 

A selected-response item in such a scenario might have fewer choices than in a conventional 
multiple-choice item. For example, a student might select between two alternatives, such as 
deciding whether a switch in a circuit should be open or closed in order to produce a particular 
outcome, but the student might also have to justify or explain the selection. In this case, the first 
part of the answer is a selected response, but it might be necessary to score the two parts of the 
item together so that the selection and justification together determine the score. In complex, 
real-world scenarios, it might be the case that there is not a “correct” selection, and in such a 
situation what matters is that the selection is justified adequately. 

An example of a selected-response item that might be part of a series of items embedded in a 
scenario-based assessment set is shown on the next page. The question was part of an ICT 
assessment about reintroducing lynx into a Canadian park overrun by hares. The assessment was 
designed for 13-year-olds. 


4-9 



Technology and Engineering Literacy Framework for the 2014 NAEP 


Park rangers noticed the problem because they've been estimating the 
number of hares in the area for the last four years. Here's what they 
found. Last year, 2002, there were about 95,000 hares. The year before 
that, 2001, there were about 80,000. In 2000, there were 25,000. And 
in 1999, there were only about 1,000 hares. 

Your task is to organize the data to see if there is a trend. 

Pick a tool to use: 
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■ 
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■ 

Presentation 


(Source: Quellmalz & Kozma, 2004) 


Other selected-response types within a scenario-based assessment set might include a task in 
which a student selects all of the options that apply from a given set of choices. Again, in a real- 
world situation there might be one “best” combination of choices but also one or more other 
combinations that are partially correct. In such a situation, it makes sense to use a scoring rubric 
that rewards different combinations of selected-response items with different scores. 

In the following item, students observe organisms interacting in an ecosystem before choosing 
all of the organisms that are consumers. The assessment was designed for grade 8. In the NAEP 
Technology and Engineering Literacy Assessment, students might choose all of the organisms in 
an ecosystem most immediately affected by a pollutant. 
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(Source: Quellmalz, Timms, & Buckley, 2009) 
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Another fonn of selected response is a “hot spot” in which a student answers by clicking on a 
spot on an image such as a map, picture, or diagram. The example below was designed for grade 
8 students as part of a set of items on a state science test. For the NAEP Technology and 
Engineering Literacy Assessment, a task might be designed that asks students to click on the 
places in the image where pollution originated in the water system. 
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(Source: Minnesota Department of Education, 2009) 


Ways of Measuring Student Responses 

The computer-based administration of the scenario-based assessments combined with the broad 
range of selected and constructed responses possible with this approach will provide many 
opportunities to measure students’ capabilities as defined in the assessment targets. The range of 
measures will be greater than those generated in a typical NAEP assessment of other subjects, so 
it is necessary to describe how all of these measures might be handled. It is helpful to think of the 
measures as falling into two categories: student direct responses, and pattern-tracking measures 
that are based on student interactions with the tools and systems portrayed in the scenarios. 

Conventional items always involve the student in a direct response. For example, after being 
presented with information in a diagram, the student is asked a text-based question and given a 
limited set of choices from which to select the best answer. Student direct responses can also be 
used in scenarios. For example, an assessment task in the scenario may have asked the student to 
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set two different values for a component of the system and observe what happens. The student 
direct response comes when, after observing the interaction with the system, the student is asked, 
for example, to compare and contrast the two outcomes and explain in a short written response 
why they happened as they did. This is a student direct response because, although the student 
interacted with the system, none of that interaction was captured to score the appropriateness of 
the student response to the item. Only the written observation and explanation are to be scored. 

One type of student direct response is selection from a set of choices — for example, multiple 
choice, checking all of the boxes that apply, or, in a scenario-based assessment, selecting an 
object or choosing a tool for the task. Other types of direct response include providing a written 
analysis of a set of results and writing a short explanation of why a selection was made in a 
scenario. 

By contrast, in pattern-tracking measures the interactions that the student engaged in may 
provide relevant evidence about whether the student possesses a skill that is an assessment target 
and that should, therefore, be captured, measured, and interpreted. For example, a student may 
have been asked to pinpoint a malfunction in a technological system, such as a leak in a lawn 
sprinkling system. In responding to that task, the student’s manipulation of the components of 
the system shows whether the student is testing the sprinkler components in a random or 
systematic way. Thus the things that the student chose to manipulate, how the student 
manipulated them, and how long it took might all be measured and interpreted in combination so 
as to provide a measure of whether the student possesses a particular skill related to 
troubleshooting. 

One type of pattern-tracking measure is the observation of patterns of action — for example, 
capturing a sequence of actions taken to determine if the correct set of actions was taken and if 
the actions were executed in the optimal order. Another pattern-tracking measure is tracking the 
manipulations that the student performs in a scenario. How, for instance, did the student change 
the features of a Web search query (for example, narrowing the topic) or vary the parameters that 
control a component of a system (for example, changing the gauge of wire mesh for bird 
protection in a model of a rooftop wind turbine) or transform an object from one fonn into 
another (for example, transfonning database entries into a graph or table for a presentation)? 
Pattern-tracking measures might be used to assess certain aspects of communication or 
collaboration skills. For example, measuring the number of times a student communicates with 
virtual team members with particular expertise can provide a measure of the efficiency of the 
student’s collaboration strategies. 

Balance of the Assessment 

To ensure an appropriate distribution of the test time, it is important to balance the different 
components of the assessment. This section discusses how this can be done. Note that “total test 
time” refers to the length of time that it would take to administer all available assessment blocks 
(see figure 3), which is the equivalent of approximately 5 hours of test time. Many students will 
be in the sample population during each administration of the test, but each of these students will 
spend approximately 50 minutes on the assessment tasks, so there must be a plan for distributing 
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each assessment block to many students. Figure 3 also shows a simple example of how 
assessment types might be grouped together. 



If all test items given to a student total test time of about 5 hours 
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Figure 3 - Potential Balance of Total Test Time 

This section discusses four separate aspects of the assessment that should be considered in 
determining an overall balance: 

• Balance by Major Assessment Area 

o Technology and Society 
o Design and Systems 

o Information and Communication Technology 

• Balance by Practice 

o Understanding Technological Principles 
o Developing Solutions and Achieving Goals 
o Communicating and Collaborating 
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• Balance by Set Type 

o Long scenarios 
o Short scenarios 
o Discrete items sets 

• Balance by Response Type 

o Selected response 
o Constructed response 

The balance required at each grade level is specified in the following sections as a percentage of 
the total test time. In other words, the percentage expresses what proportion of the total amount 
of testing time — as represented in the total item pool shown at the top of the diagram in table 
4.1 — would be allocated. Since a student is assigned to take only one of the groups of item sets 
shown in table 4.1, the percentages do not necessarily represent the distribution of time in any 
single student’s test session. It should also be noted that the percentage of testing time 
distributions shown in tables 4.1 through 4.4 are to be regarded as targets, and it is understood 
that percentages in the actual assessment may vary slightly from the target percentages. 

The balances shown in the following tables 4.1 through 4.4 were detennined after careful 
consideration and deliberation by both the planning committee and the steering committee 
members. They reflect the expert judgment of the committees after they took into account how 
the knowledge and skills in the area of technology and engineering literacy are taught at different 
grade levels and how they develop over time. The percentages shown in the tables were derived 
from a process in which planning committee members made individual judgments, the group 
results were tabulated, and then discussion followed to come to a consensus that is represented in 
tables 4.1 through 4.4. 

Assessment Balance by Major Assessment Area 

Table 4.1 shows the balance by major assessment area at each of the 3 grade levels. At fourth 
grade there is an emphasis on ICT because the focus of technology and engineering literacy 
instruction at that grade is on using common infonnation and communication technologies. At 
eighth grade the balance is weighted to Design and Systems because in middle school there is 
more emphasis on systems, and there is slightly less time spent on ICT than in the early grades. 
At 12th grade the balance is slightly weighted to Design and Systems and to ICT. 
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Table 4.1 Assessment balance by major assessment areas and grades 


" \ Grade 

Major 

Assessment Area 

Grade 4 

(% total test time) 

Grade 8 

(% total test time) 

Grade 12 
(% total test time) 

Technology and Society 

25 

25 

30 

Design and Systems 

30 

40 

35 

Information and 
Communication 
Technology (ICT) 

45 

35 

35 


Assessment Balance by Practice 

The balance of the assessment by crosscutting practices across the 3 grade levels is shown in 
table 4.2. At all grades the balance of total test time is: 

• Understanding Technological Principles - 30 percent 

• Developing Solutions and Achieving Goals - 40 percent 

• Communicating and Collaborating - 30 percent 

The rationale for a slight emphasis on the practice of Developing Solutions and Achieving Goals 
is that it is important for students to be able to use their knowledge of technological principles in 
developing solutions to problems. 
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Table 4.2 Assessment balance by practices and grades 


Grade 

Practice 

Grade 4 

(% total test time) 

Grade 8 

(% total test time) 

Grade 12 
(% total test time) 

Understanding 
Technological Principles 

30 

30 

30 

Developing Solutions and 
Achieving Goals 

40 

40 

40 

Collaborating and 
Communicating 

30 

30 

30 


Assessment Balance by Set Type 

In addition to ensuring a balance across the content of the assessment, table 4.3 specifies how the 
total amount of testing time should be balanced across the 3 types of assessment sets — long 
scenario, short scenario, and discrete item sets. There is an emphasis on the scenarios because 
they allow for a wide range of interactive tasks that are well-suited to assessing the types of 
practices that students need to apply in practical investigations. The 20 percent of discrete items 
allow for more independent testing of knowledge and skills that help to maintain test reliability. 

Table 4.3 Assessment balance by set types and grades 


— Grade 

Set Type 

Grade 4 

(% total test time) 

Grade 8 

(% total test time) 

Grade 12 
(% total test time) 

Long Scenarios 
(Approx. 25 minutes) 

40 

40 

40 

Short Scenarios 
(Approx. 12-15 minutes) 

40 

40 

40 

Discrete items 
(Approx. 1-2 minutes per 
item; approx. 12-15 minutes 
per block) 

20 

20 

20 
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Assessment Balance by Response Type 

Table 4.4 specifies the balance of assessment response types across the total testing time. Across 
all grade levels, there will be an emphasis on constructed-response items because the more 
complex tasks involved in scenarios demand that students are involved in activities that create 
products rather than simply select among choices of responses. Since the balance of types of 
assessment types specified in table 4.3 is weighted toward scenarios, there is a corresponding 
weighting toward constructed responses, although selected responses also form an important part 
of the assessment. 

Table 4.4 Assessment balance by response types and grades 


^''\^^^Grade 

Grade 4 

Grade 8 

Grade 12 

Response Type 

(% total test time) 

(% total test time) 

(% total test time) 

Selected Response 

40 

40 

40 

Constructed Response 

60 

60 

60 


Assessment Design and Student Diversity 

Students should have the opportunity to demonstrate their knowledge of the concepts and ideas 
that the NAEP Technology and Engineering Literacy Assessment is intended to measure. The 
assessment needs to be responsive to the challenges that stem from an increasingly diverse 
student population in the nation, the inclusion of all types of students in the general curriculum, 
and an increased emphasis and commitment to serve and be accountable for all students. NAEP 
should strive to develop assessments that allow for the participation of the widest possible range 
of students so that interpretation of scores of all who participate leads to valid inferences about 
the levels of their performance. 

As the NAEP Technology and Engineering Literacy Assessment will be computer-delivered and 
will contain a large proportion of scenario-based assessment items, great attention needs to be 
paid to ensuring that the assessments are valid and accessible to a wide range of students. Two 
types of populations need to be considered in order to properly address student diversity in the 
design of the assessment, English language learners — students who are developing English as 
their second language — and students with disabilities. Both populations are tremendously 
heterogeneous: the former, because of the wide range of students’ first languages, cultural 
influences, and stages in English development; the latter, because of the wide range of 
disabilities — from physical to sensory to cognitive. Whereas both populations may have common 
sets of needs, some design issues may be particularly relevant to one or the other. Both 
populations should be considered carefully in the design of the NAEP Technology and 
Engineering Literacy Assessment. 
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To effectively include English language learners and students with disabilities in the technology 
and engineering literacy assessments, two areas that need to be addressed include the 
participation of students in the pilot stages of test development and the use of testing 
accommodations. These aspects are discussed in more detail in the Assessment and Item 
Specifications for the 2014 NAEP Technology and Engineering Literacy Assessmen t. 

Pilot Student Samples 

The term “inclusion” does not refer solely to the participation of students with special needs in 
large-scale testing. It also refers to the notion that, to attain equity in testing, appropriate actions 
should be taken to ensure that these students participate in the process of assessment 
development. 

Issues related to students with special needs should not be addressed only at the end of the 
process of assessment development, guided by the erroneous belief that testing accommodations 
provided during test administration will ensure equitable testing. Rather, English language 
learners and students with disabilities should be included in ah of the assessment tryout stages in 
which infonnation is collected from pilot students about the ways in which they interpret 
stimulus materials and test items and the difficulties they experience in providing their responses. 
Even when assessments are to be administered only in English, English language learners can 
provide test developers with valuable infonnation for improving the assessment (for example, 
the wording of its items). Also, the infonnation provided by students with disabilities can help to 
identify and address usability issues that are relevant to administering the assessment to ah 
students. 

A potential challenge in the testing of students with special needs is the fact that the critical 
characteristics of these students are not always carefully identified (Solano-Flores, 2009). As a 
result, they are not properly represented in the samples of students who participate in the pilot 
stages of assessment development. The use of very small samples of English language learners 
and students with disabilities makes it difficult for test developers to properly address the 
heterogeneity of these groups. Thus, preparation for the operational assessment should address 
the diversity within the population of English language learners (for example, diverse first 
language backgrounds and different levels of English development) and within the population of 
students with disabilities (for example, diverse types of disabilities and various levels of severity 
of disability). 

Universal Design 

The items in the NAEP Technology and Engineering Literacy Assessment that make use of the 
multimedia capabilities of the computer offer a chance to more easily reach a wide range of 
students than traditional paper-and-pencil items allow. Universal design is a concept that started 
in architecture and has been applied to assessment by identifying the relevant essential elements 
for assessment (Thompson, Thurlow, & Malouf, 2004). Since its early explication, the concept 
has been applied to instruction as universal design for learning (Orkwis & McLane, 1998; Rose 
& Meyer, 2002). At the time of the writing of this framework, NAEP is considering how the 
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principles of universal design might be applied in its assessments, but no policy has yet been 
adopted. 

The Universal Design for Computer-Based Testing (UD-CBT) framework and a detailed set of 
UD-CBT guidelines (Dolan et al, 2007) specifically address the design of novel computer- 
delivered assessments such as the one that will be developed for the NAEP Technology and 
Engineering Literacy Framework. The UD-CBT represents, at the time of publication of the 
NAEP Technology and Engineering Literacy Framework, a useful guideline for developing 
interactive scenario-based assessments. However, developers of the assessment will need to pay 
attention to developments in the field as a greater number of innovative assessments are 
developed and more information is available about how to apply the principles of universal 
design to them. 

Accommodations 

NAEP strives to assess all students selected by its sampling process. Rigorous criteria are applied 
to maximize the number of English language learners and students with disabilities included in 
NAEP assessments. Participating students with special needs are pennitted to use 
accommodations, as stated in current NAEP policy: 

All special-needs students may use the same accommodations in NAEP 
assessments that they use in their usual classroom testing unless the 
accommodation would make it impossible to measure the ability, skill, or 
proficiency being assessed, or the accommodation is not possible for the NAEP 
program to administer (NCES, 2005, Current Policy section, ]f 4). 

To meet this commitment, it will be necessary for the assessment delivery system to include tool 
options that allow the students to benefit from the sorts of accommodations that they need. An 
example of these tool options is screen magnification, which can benefit blind or low-vision 
students. Another example is text-to-speech, which may also benefit blind or low-vision students 
as well as English language learners whose proficiency in English is better in the listening mode 
than in the reading mode. 

To make accommodations more likely to successfully serve students with special needs, an 
important fact should be taken into account in assessment design. Each English language learner 
is unique as to his or her level of English proficiency in the listening, speaking, reading, and 
writing modes and each student with disabilities is unique as to the kind and severity of his or her 
disabilities. As a consequence, what works for one student does not necessarily work for another 
student within the same group of students with special needs. Designing an assessment delivery 
system that is capable of providing all NAEP-authorized accommodations and that allows 
selection of the set of accommodations that best meets the needs of each student is critical to 
properly meeting the goals of inclusion in the NAEP Technology and Engineering Literacy 
Assessment. 
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Chapter Five: Reporting Results of the NAEP 
Technology and Engineering Literacy Assessment 


Introduction 

The purpose of this chapter is to explain how the results of the NAEP Technology and 
Engineering Literacy Assessment will be reported. A probe will be administered in 2014 as a 
trial of the NAEP Technology and Engineering Literacy Assessment. In the NAEP context, a 
probe is a smaller-scale, focused assessment on a timely topic that explores a particular question or 
issue and may be limited to particular grades. The description in this chapter is for the initial probe 
assessment, but can be applied to future administrations of the assessment beyond that first 
assessment. Key sections of the chapter are as follows: 

• How NAEP Results Are Reported; 

• Reporting Scale Scores and Achievement Levels; 

• Reporting Background Variables; and 

• Uses of NAEP Reporting. 

How NAEP Results Are Reported 

The National Assessment of Educational Progress provides the only nationally representative 
report on student achievement in a variety of subjects. Starting in 2013, NAEP results will be 
published mainly online, on a website that will give the public access to an interactive version of 
the report card. An executive summary will be available in print to accompany the online data. 
The online resource provides detailed information on the nature of the assessment, the 
demographics of the students who participate, and the assessment results. 

The Nation’s Report Card includes information on the performance of various subgroups of 
students at the national level. Subgroups for NAEP include: 

• Gender; 

• Race/ethnicity; 

• Eligibility for free/reduced-price lunch; 

• Students with disabilities; and 

• English language learners. 

Detailed data on NAEP results, demographic variables, and subject-specific background 
information are available via the NAEP Data Explorer on the website. Additional restricted data 
are available for scholarly research, subject to National Center for Education Statistics (NCES) 
licensing procedures. 

The Nation’s Report Card provides results on the performance of students in public schools in 
various states as well as in the NAEP Trial Urban Districts. The Trial Urban District Assessment 
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was initiated in 2002 to report on the achievement of public school students in large urban 
districts. The NAEP Technology and Engineering Literacy Assessment will not be administered 
as part of the Trial Urban District Assessment program. Results will be reported only at the 
national level. 

Reporting Scale Scores and Achievement Levels 

Results of the NAEP Technology and Engineering Literacy Assessment will be reported in terms 
of percentages of students who attain each of the three achievement levels, Basic, Proficient, and 
Advanced, discussed below. The NAEP Technology and Engineering Literacy Assessment is an 
assessment of overall achievement, not a tool for diagnosing the needs of individuals or groups 
of students. Reported scores are always at the aggregate level. By law and by design, scores are 
not produced for individual schools or students. Results will be reported for the nation as a whole 
as well as for regions of the nation. The NAEP Technology and Engineering Literacy 
Assessment will not provide results for individual states, as the student samples will be drawn to 
report at the national level only. 

The project committees have recommended that the results of the assessment be reported in 
tenns of three subscores, each of them reflecting performance in one of the three main areas of 
technology and engineering literacy: Technology and Society, Design and Systems, and 
Infonnation and Communication Technology. An overall composite score will also be reported. 
At this time technology instruction in most K-12 schools focuses on one or another of the three 
areas rather than a fusion of the three, so a composite score can be expected to have less 
relevance than the scores from the three areas. 

Reporting on achievement levels is the primary way in which NAEP results reach the general 
public and policymakers. Achievement level results indicate the degree to which student 
performance meets the standards set for what students should know and be able to do at the 
Basic, Proficient, and Advanced levels. Definitions of achievement levels articulate expectations 
of performance at each grade level. They are reported as percentages of students within each 
achievement level range, as well as the percentage of students at or above Basic and at or above 
Proficient ranges. Results for students not reaching the Basic achievement level are reported as 
below Basic. Results are also reported for subgroups of students using demographic data and 
background variables specific to the NAEP Technology and Engineering Literacy Assessment. 
An individual student’s performance cannot be reported based on NAEP results. 

Table 5.1 displays the Governing Board’s generic policy definitions for Basic, Proficient, and 
Advanced achievement that pertain to all NAEP subjects and grades. 
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Table 5.1 Generic achievement level policy definitions for NAEP 


Achievement Level 

Definition 

Advanced 

This level signifies superior performance. 

Proficient 

This level represents solid academic performance for each grade assessed. Students 
reaching this level have demonstrated competency over challenging subject matter, 
including subject-matter knowledge, application of such knowledge to real-world 
situations, and analytical skills appropriate to the subject matter. 

Basic 

This level denotes partial mastery of prerequisite knowledge and skills that are 
fundamental for proficient work at each grade. 


There are three components to the NAEP achievement levels: 

• Achievement level definitions; 

• Cut scores; and 

• Examples of students’ responses. 

Achievement Level Definitions 

Since 1990, the Governing Board has used student achievement levels for reporting results on 
NAEP assessments. The achievement levels represent an informed judgment of “how good is 
good enough” in the various subjects that are assessed. Generic policy definitions for 
achievement at the Basic, Proficient, and Advanced levels describe in very general terms what 
students at each grade level should know and be able to do on the assessment. Technology and 
engineering literacy achievement levels specific to the 2014 NAEP Technology and Engineering 
Literacy Framework will be developed to elaborate the generic policy definitions of Basic, 
Proficient, and Advanced achievement for NAEP assessments. Preliminary achievement level 
definitions have been developed for each of the three areas to be reported separately in the 
assessment and they will be used to guide item development and initial stages of standard setting 
for the 2014 NAEP Technology and Engineering Literacy Assessment. (See appendix G for 
these preliminary definitions.) 

The preliminary achievement level definitions will be revised when actual student responses 
have been collected and analyzed. The Governing Board will convene panels of experts to 
examine the preliminary achievement level definitions and to recommend final achievement 
level definitions for each grade level. A broadly representative panel of exceptional teachers, 
educators, and professionals will then be convened to engage in a standard-setting process to 
detennine the cut scores that correspond to these achievement level definitions. The panelists 
will be trained and will engage in a series of discussions designed to ensure informed judgments 
about mapping cut scores to the assessment. 

Cut Scores 

Cut scores, the second component of reporting on achievement levels, represent the minimum 
score required for performance at each NAEP achievement level. Cut scores are reported along 
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with the percentage of students who scored at or above the cut score. As described in chapter 
four, the assessment design for the 2014 NAEP Technology and Engineering Literacy 
Assessment incorporates scores from selected responses and written responses, as well as 
measures of the patterns of action a student takes in problem-solving. Selected responses in 
which there is a single best answer will be scored as correct or incorrect and written responses 
will be scored using a rubric that rewards answers according to their match to descriptions in the 
rubric. The pattern- tracking will be evaluated by comparing the pattern of action against a set of 
possible patterns, and students will get more credit for a course of action that is optimal than for 
alternative patterns of action. Scores can then be combined to produce overall scores so that cut 
score decisions can be made. 

Examples of Students ’ Responses 

The third component of achievement level reporting includes examples of student responses on 
released tasks from the NAEP Technology and Engineering Literacy Assessment. These 
examples provide illustrations of student skills within each level of achievement for each of the 
three areas and will be developed after the first administration of the assessment. Example 
responses will be annotated to explain the score for the response. 

Reporting Background Variables 

Background data on students, teachers, and schools are needed to fulfill the statutory requirement 
that NAEP include information, whenever feasible, for groups identified in the first section of 
this chapter (for example, gender, race/ethnicity). Therefore, students, teachers, and school 
administrators participating in NAEP are asked to respond to questionnaires designed to gather 
demographic infonnation. Information is also gathered from non-NAEP sources, such as state, 
district, or school records. For the 2014 NAEP Technology and Engineering Literacy 
Assessment, only student and school information will be collected, since many students will not 
have taken a separate course in technology and engineering literacy taught by a specific teacher. 

In addition to demographic infonnation, background questionnaires include questions about 
variables related to opportunities to leam and achievement in technology and engineering 
literacy. The variables are selected to be of topical interest, to be timely, and to be directly 
related to academic achievement and current trends and issues in technology and engineering 
literacy. Questions do not solicit information about personal topics or information irrelevant to 
the collection of data on technology and engineering literacy achievement. 

The important components of NAEP reporting are summarized in table 5.2. Recommendations 
for background variables for the 2014 NAEP Technology and Engineering Literacy Assessment 
are presented in the separate background variables document. 


5-4 



Technology and Engineering Literacy Framework for the 2014 NAEP 


Table 5.2 Summary of NAEP reporting components 




Key Characteristics 

OX 

How 

Starting in 2013, elements released to the public will include: 

a 

Information 

• Results published mainly online with an interactive report card 

o 

a 

Is Reported 

• Dedicated website: www.nationsreportcard.gov 

a> 

pi 


• Performance of various subgroups at the national level published in 

Pm 


print and online 

◄ 

Z 

What Is 

NAEP data are reported by: 

© 

Reported 

• Percentage of students attaining achievement levels 

© 


• Scale scores 

© 

= 

0 


• Sample responses to illustrate achievement level definitions 

a 

£ 

© 

What 

T n fn v m ft t i n n 

Types of background variables distributed to students and schools: 

o 

Is Gathered 

• These are presented in the separate background variables document. 


Uses of NAEP Reporting 

The information available from results of the probe for the 2014 NAEP Technology and 
Engineering Literacy Assessment will provide important data that can be used throughout the 
tenure of the framework. The results of the probe will begin the trend line for the new 
assessment, and policymakers, educators, and the public can use data from the assessments as a 
tool for monitoring certain aspects of student achievement in technology and engineering literacy 
over time. NAEP reports from any subsequent administrations of the assessment will compare 
student performance in the three areas of Technology and Society, Design and Systems, and ICT 
among groups of students within the same grade. Long-tenn achievement trends (for example, 
the comparison of score perfonnance with previous administrations) can also be reported starting 
with the second administration. 

The scores from the assessment will be of value and interest not just to technology and 
engineering teachers but to a broad range of educators. As discussed earlier, many different types 
of teachers are involved in teaching their students about technology and its applications in grades 
K-12, from those specializing in science, math, and engineering, to those teaching social 
sciences, humanities, and the arts as well as members of cross-disciplinary teams. 

Because the NAEP Technology and Engineering Literacy Assessment will measure some 
technology and engineering literacy experiences but not all, there will be limitations to the range 
and scope of information it can produce. NAEP publishes data on student perfonnance in 
relation to various achievement levels and demographic subgroups; the infonnation reported 
does not evaluate results or provide conclusive statements about the level of achievement among 
the nation’s K-12 students. Furthermore, the NAEP Technology and Engineering Literacy 
Assessment is not designed to infonn instruction — to guide how technology and engineering 
literacy is taught — but only to measure the performance of a representative sample of the 
American student population at the designated grade within the assessment context outlined in 
this framework. 
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Appendixes 


Appendix A: Glossary of Abbreviations, Words, and Terms Used in the 
Framework 


The glossary is divided into five sections. The first section presents acronyms and abbreviations 
used in the framework, followed by sections on basic framework terminology, assessment terms, 
and terms related to education content and pedagogy. The final section presents terms specific to 
the three major assessment areas of technology and engineering literacy: Technology and 
Society, Design and Systems, and Information and Communication Technology. Relevant terms 
and definitions from each area are included, and they are defined within the context of the 
framework. 


Acronyms and Abbreviations for Associations, Educational Organizations, or Reports 


AAAS 

ccsso 

IBO 

ISTE 

ITEEA 

NAE 

NAEP 

NCES 

NETS*S 

NRC 

NSTA 

OECD 

PISA 

SETDA 

STEM 

TIMSS 


American Association for the Advancement of Science 

Council of Chief State School Officers 

International Baccalaureate Organization 

International Society for Technology in Education 

International Technology and Engineering Educators Association 

National Academy of Engineering 

National Assessment of Educational Progress 

National Center for Education Statistics 

National Educational Technology Standards for Students 

National Research Council 

National Science Teachers Association 

Organisation for Economic Co-operation and Development 

Programme for International Student Assessment 

State Educational Technology Directors Association 

Science, Technology, Engineering, and Mathematics 

Trends in International Mathematics and Science Study 


Basic Framework Terminology 

Technology Any modification of the natural or designed world done to fulfill 

human needs or desires. 

Engineering A systematic and often iterative approach to designing objects, 

processes, and systems to meet human needs and wants. 

Technology and Capacity to use, understand, and evaluate technology as 

engineering literacy well as to understand technological principles and strategies 

needed to develop solutions and achieve goals. It encompasses the 
three areas of Technology and Society, Design and Systems, and 
Infonnation and Communication Technology. 
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Educational technology 
Technology educators 
Technology education 

Framework 

Technological processes 
Technological principles 

Technological practices 

Assessment Terms 

Advanced achievement level 

Assessment areas or targets 
Assessment balance 
Assessment specifications 


Various types of equipment, tools, and processes used as aids in 
teaching and learning. 

All those whose teaching responsibilities include imparting the 
knowledge, capabilities, and skills described in this framework. 

The knowledge and skills taught to students in the three areas of 
Technology and Society, Design and Systems, and Information and 
Communication Technology. 

A NAEP framework is a document that defines the parameters of a 
NAEP assessment. It guides the test makers in developing an 
assessment. 

Procedures using technological knowledge, tools, and skills to 
develop solutions and achieve goals. 

Sets of foundational and fundamental assumptions that underlie 
each of the three areas of technology and engineering literacy 
defined in this framework. 

Types of thinking and reasoning that students are expected to 
demonstrate when responding to an assessment item. The 
framework specifies three practices: understanding technological 
principles; developing solutions and achieving goals; and 
communicating and collaboration. 


The highest of NAEP’s three levels of performance. This level 
signifies superior performance. 

The three assessment foci of this framework on technological 
literary: Technology and Society, Design and Systems, and 
Infonnation and Communication Technology. 

Appropriate distribution of items according to major assessment 
area, technological practice, assessment set type, and response 
type. 

Assessment requirements that framework developers give to test 
developers. These include, for example, the foci of the assessment, 
the number and types of items, the specific areas to be assessed, 
the accommodations for students with disabilities, etc. 
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Background variables 
Basic achievement level 
Constructed response 

Cut scores 
Discrete item set 
Item 

Item specifications 
Probe (noun) 

Proficient achievement level 

Response type 
Scale scores 


Demographic and contextual data related to the NAEP assessment 
gathered through questionnaires usually completed by school 
administrators, teachers, and students. 

The lowest of NAEP’s three levels of performance. This level 
denotes partial mastery of prerequisite knowledge and skills that 
are fundamental for proficient work at each grade. 

Items in which the student “constructs” the response 
rather than choosing a response from a limited number of 
alternatives. Constructed responses may be short (students supply a 
word or short sentence) or extended (students complete a task). 

The minimum score required for performance at each NAEP 
achievement level. 

A group of questions that include conventional selected-response 
items and short constructed-response items. 

A single question or set of instructions. 

Assessment requirements that framework developers give to test 
developers; for example, the number and types of items to be 
included. 

A smaller-scale, focused assessment on a timely topic that explores 
a particular question or issue and may be limited to particular 
grades. 

The middle of NAEP’s three levels of performance. This level 
represents solid academic performance for each grade assessed. 
Students reaching this level have demonstrated competency over 
challenging subject matter, including subject matter knowledge, 
application of such knowledge to practical situations, and 
analytical skills appropriate to the subject matter. 

The activity an item asks a student to perfonn when responding. In 
this assessment there are three item response types: short 
constructed response, long constructed response, and selected 
response. 

Scores that allow for comparison of students’ perfonnances on 
different administrations of a test. For example, students’ scores 
might be converted to a score on a scale that ranges from 0 to 500 
points. 
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Scenario-based assessment In the context of this framework, scenarios are interactive 

computer tasks that constitute the bulk of the items. Scenarios may 
be short or long, depending on what they require the student to do. 

Selected response A type of item in which students read a question and choose the 

best answer from a set of options. 

Universal design for Guidelines for ensuring that the largest number of disabled 

assessment students and English language learners participate in an 

assessment. 

Education Content and Pedagogy 

Academic problem An assigned task that a teacher may give to a student. 


Collaboration 


Diffuse curriculum 
Disaggregation 

Equity 

Fluency 
Habits of mind 
Literacy 

Sequential curriculum 


To work together with other individuals. However, for this NAEP 
assessment, it will mean using contemporary technologies to work 
with virtual (computer-generated) individuals to solve problems or 
achieve goals. 

Curriculum without a clear scope, sequence, and series of courses. 

Separation into component parts, such as the breaking down of 
achievement data by racial/ethnic subgroup. 

Fair access to opportunities to learn that are based on need rather 
than on some arbitrary factor. 

A smooth and easy flow of knowledge and skills. 

Customary ways of thinking and acting. 

The capacity to use, understand, and evaluate a body of knowledge 
and skills as well as to apply concepts and processes to solve 
problems and reach one’s goals. 

Curriculum that has a scope, sequence, and a series of courses. 


Area-Specific Terms 

Technology and Society 

Artifacts Products and items that a society’s population develops, uses, and 

updates to meet needs and wants. 
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Human-made 
Modeling and simulation 

Natural world 
Practical problem 
Product life cycle 

Regulating technologies 
Technological inequalities 
Trade-off 

Design and Systems 
Constraint 

Criteria 

Engineering design method 
(sometimes called 
technological design) 


Term describing an artifact that has been designed and developed 
by means that are outside the boundaries and capabilities of the 
natural world. 

Utilizing technology to analyze and test the possible effects, 
impacts, and trade-offs that are associated with a new 
technological innovation to evaluate efficiency, discover potential 
problems, and develop workable solutions. 

Plants, animals, water, and other organisms and elements that exist 
without contributions from humans. 

A situation that a person may encounter in everyday life that 
requires a solution. 

The span of time that an artifact is commissioned to satisfy a 
societal need that can start from the point of design, manipulation 
of raw materials, and manufacturing processes, to eventual 
obsolescence and disposal. 

Technological innovations that are responsible for contributing to 
the protection of natural resources in areas such as transportation, 
energy, and waste disposal. 

Instances where countries and societies use antiquated 
technologies due to economic circumstances or cultural 
preferences. 

A decision where complete awareness of both the advantages and 
disadvantages of the result are explored and the impacts of both are 
taken into consideration. 


A boundary, limit, or restriction, such as time, money, or 
resources, in the requirements for a project. 

Characteristics (or specifications) of a successful solution, such as 
a desired function or a particular level of efficiency. 

An iterative, systematic process for solving problems that 
involves creativity, experience, and accumulated 
disciplinary knowledge. 


A-5 



Technology and Engineering Literacy Framework for the 2014 NAEP 

Life cycle 

Important phases in the development of a system from initial 
concept through design, testing, use, and maintenance to 
retirement. 

Optimization 

Finding the best possible solution when some criterion or 
constraint is identified as the most important and others are given 
less weight. 

Problem-solving 

The cognitive process of finding answers to questions and 
solutions to undesired situations. 

Prototype 

First version, or generation, of an entity created from a particular 
design plan using the engineering design method. 

Requirements 

Combination of criteria and constraints for a given project. 

Reverse engineering 

Disassembling an item in a systematic way to understand how it 
works, usually so it can be repaired, copied, or improved. 

Systems thinking 

Way of investigating or thinking about a system using a set of 
principles. 

Troubleshooting 

Systematic method of dealing with failures. 

Information and Communication Technology 

Digital assistants 

Also called personal digital assistants (PDAs), devices used as 
organizers to enter and store data such as addresses, expenses, or 
calendar items. Some are capable of being used as hand-held 
computers and may also have Internet capabilities. 

Digital models 

An electronic representation of a system. 

Digital tools 

Any technology that stores and transmits data electronically. 

Fair use 

A condition under U.S. copyright law that permits limited use of 
copyrighted material without procuring permission from the 
copyright holder. 

Geographical information 
system 

Any system that gathers, saves, evaluates, and presents data 
related to geographic locations. A common example is a GPS, used 
to obtain driving directions. 

Information and 
Communication 
Technologies 

Technologies used to access, gather, store, analyze, and 
report information. 
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Information (or digital) 
literacy 

Interactive whiteboards 

Media literacy 

Media player 
Mobile wireless devices 

Wiki 


Skills needed to access, evaluate, and use information from 
a variety of sources. 

An interactive display system that connects to a projector and a 
computer. Using special software, it is possible to project the 
computer’s desktop and then control the computer using a stylus, 
personal response system, or even a finger. 

The capacity to access and evaluate messages created using a 
variety of media, such as advertisements, commercials, or 
speeches, etc. It also refers to the skills required to develop and 
communicate a message using media. 

Software used on a computer to manage and play video or audio 
files or to view digital images. It can also be hand-held hardware 
that provides the same functions and is also used to store files. 

Small, lightweight hardware, often called hand-held technology, 
that has the capability to connect wirelessly to the Internet. 
Examples include hand-held computers, smartphones, and 
netbooks. 

A website that allows users to work collaboratively to view, edit, 
and add information. One of the best-known examples of this type 
of site is Wikipedia, a collaboratively written encyclopedia. 


A-7 



Technology and Engineering Literacy Framework for the 2014 NAEP 


Appendix B: Steering Committee Guidelines* 

December 18, 2008 

Provisional Approval: September 16, 2009 


Introduction 

The steering committee guidelines that follow are framed, in part, by the Issues and 
Recommendations paper prepared by Sharif Shakrani (Michigan State University) and Greg 
Pearson (National Academy of Engineering) for the National Assessment Governing Board. The 
guidelines were developed over a two-day period (December 17-18, 2008) by the steering 
committee for the 2012 NAEP Technological Literacy Assessment Project. The steering 
committee presented these guidelines to the planning committee in a joint session on December 
18, 2008. The guidelines were revised and reorganized according to decisions made during the 
steering committee’s meeting on March 1 1-12, 2009, and again shared with the planning 
committee. In addition, the steering committee sought input from leaders in education, industry, 
business, engineering, research, and allied organizations (including the International Society for 
Technology in Education [ISTE], the International Technology and Engineering Educators 
Association [ITEEA], and the Partnership for 21st Century Skills); feedback resulting from these 
reviews served to further refine the guidelines. 

Suggested Definition for Technological Literacy 

The steering committee has identified various elements that illustrate the knowledge, ways of 
thinking and acting, and capabilities that define technological literacy. Technological literacy, as 
viewed by the steering committee, is the capacity to use, understand, and evaluate technology as 
well as to understand technological principles and strategies needed to develop solutions and 
achieve goals. It encompasses the three areas of Technology and Society, Design and Systems, 
and Infonnation and Communication Technology. 

Recommended Grade Level to Be Assessed 

The National Assessment Governing Board requested that the two committees recommend to the 
Board at what grade(s) the national probe should be conducted in 2012. Hence, the steering and 
planning committees of the NAEP Technological Literacy Framework Development Project, in a 
joint session on March 11, 2009, discussed the matter. On the basis of this discussion, the two 
committees recommend to the Governing Board that the proposed 2012 NAEP Technological 
Literacy probe be administered at grade 8. If funding is available for an additional probe, the 
committees recommend that it be given at grade 12. The lowest priority would be for a probe at 
grade 4. The rationale for choosing grade 8 included: 

• Students are cognitively mature and are more likely to have taken a technology-related 
course or curriculum units. 


These guidelines reflect the recommendations of the steering committee as of September 2009 and do not include 
the change in title and year of the assessment approved by the National Assessment Governing Board in March, 
2010 . 
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• This is the last grade before student dropout rates increase. 

• Grade 8 is aligned with the Trends in International Mathematics and Science Study 
(TIMSS) and the Programme for International Student Assessment (PISA) assessments, 
allowing for more data analysis. 

• Differences in performance by gender are less pronounced than at grade 12. 

• There is more opportunity to measure impact from schooling than at grade 4. 

• Students may be better equipped to take a computer-based test than at grade 4. 

• Grade 8 is specifically targeted in No Child Left Behind, which requires that every 
student is technologically literate by the time the student finishes the eighth grade. 

Guiding Principles 

The assessment shall consist of technological content areas making up the scale scores to be 
reported and technological practices that characterize the field. 

1 . There are two content areas that must be assessed and where the data must be reported out as 
subscales: 


a. Design and Systems 

b. Infonnation and Communication 
Technology (ICT) 

If possible, the following area should be explored 
as a third content area of focus: 

c. Technology and Society 

See the addendum for examples of targets to be 
included in the three content areas. 

2. Suggested technological practices are listed 
below: 

a. Understanding Technological Principles 

b. Developing Solutions and Achieving 
Goals 

c. Communicating and Collaborating 


The relationships of these practices to the content areas and to each other are reflected in figure 
B-l. 
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Figure B-l: The relationship between 
major assessment content areas (blue) and 
technological practices (yellow). 


3. Content and context for the assessment should be informed by existing state standards and 
assessments, national (for example, International Society for Technology in Education 
[ISTE], International Technology and Engineering Educators Association [ITEEA]) and 
international standards, and research (National Academy of Engineering, National Research 
Council). Examples from industry, federal agencies responsible for carrying out STEM- 
based research (NASA, National Oceanic and Atmospheric Administration [NOAA], U.S. 
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Department of Agriculture, National Science Foundation [NSF], U.S. Department of Energy, 
U.S. Department of Transportation, etc.), and science and technology museums might also be 
collected for building assessment ideas. 

4. The assessment should have tasks that are applied to real-world contexts and should be 
scaffolded to ensure that reliance on students’ prior knowledge of specific technology 
systems will be minimized. The focus should be on concepts and not on specific vocational 
skills or technologies. 

5. “Life situations” and local and contemporaneous conditions should be used as a way to 
confer relevance to each grade level. Content and context of the assessment items should 
have relevance and meaning to the learner. If possible, options should be generated in the 
assessment representing different situations to remove bias due to the background of the 
student. 

6. The planning committee must develop examples of items and tasks that measure only 
knowledge and practices that are assessable. The examples should not be limited, to multiple 
choice, but should illustrate, if possible, different styles and technological tools. 

7. Considerable work has already been put into the development of state standards for 
technology, ICT fluency, and engineering. These existing state standards and assessments 
should inform the development of the NAEP framework, but should not limit the framework. 
The planning committee should consider differences in how technological literacy is defined 
and treated in state standards, whether as discrete standards or as part of core content 
standards, e.g. mathematics, science, and language arts. Attention should also be paid to the 
confusion around Educational Technology versus Technological Education. There are 
numerous studies that specify the nature of technological literacy within the standards of 
various states. States each have their own definition of technological literacy, which may 
vary from the definition used in the NAEP framework. 

8. To avoid the potential problem of obsolescence, the 2012 Technological Literacy Assessment 
should focus on a broad base of knowledge and skills, not on specific technologies that may 
change. For example, specific communication technologies in use today (Internet-connected 
multimedia, smartphones and PDAs) would not have been familiar to students a decade ago, 
and will most likely be obsolete a decade from now. 

9. The assessment should use innovative computer-based assessment strategies that are 
informed by research on learning and are related to the assessment target (for example, 
problem-solving). These strategies should also reflect existing state computer-based 
assessments and the computer-based assessment aspects of the 2009 NAEP Science 
Framework. To effectively integrate innovative computer-based assessment strategies in the 
NAEP Technological Literacy Framework, the planning committee and/or assessment 
developers need to know the affordances and constraints of various technologies, how 
particular ones could support assessment goals, and how to use them. Specific examples of 
tools available for assessment tasks include probeware, presentation software, authoring 
tools, electronic whiteboards, drawing software, and typing tutors. 
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10. Computer-based assessment strategies should be informed by what is known about all 
learners, including those with special needs (English language learners and students with 
disabilities). These strategies should also be informed by the ways students currently use 
computers to access, process, and utilize information (Web 2.0, social networking, etc.). 
Effective assessment design incorporates current understandings of how people leam, how 
experts organize infonnation, and the skills of effective learners. Development of the 
framework and item specifications should draw on expertise in engineering and technology 
to suggest types of problems that might be adapted to computer-based testing. The kinds of 
technological tools scientists and engineers use, such as simulations and visualizations, 
should be considered for use in the technological literacy assessment. 
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Addendum 

The steering committee formulated the following list of targets they thought should be included 
in each of the content areas. Although they consider these sets of targets to be necessary, they 
agree that the specified targets may not be sufficient. In addition, the committee recognized that 
guidelines are normally “big ideas” and that this list contains detail in some areas and not in 
others. In the areas where detail appears, the committee considers these details sufficiently 
important to enumerate them for the planning committee’s consideration on individual targets. 


Major Assessment Areas 

Targets 

Technology and Society 

a. Where does technology come from? 

b. Why is it created? 

c. How is society affected by scientific and technological 
concepts and capabilities? 

d. How technology affects the environment and vice versa 

e. Assessing consequences of technology (intended and 
unintended) 

f. Green technology 

g. Digital citizenship 

h. Safety 

i. Ethics 

j . Responsible use of technology 

Design and Systems 

a. Artifacts are the products of technological systems that 
meet a need or solve a problem 

b. Identifying the technological dimensions of an artifact 

c. Constraints 

d. Models (and their importance and limitations) 

e. Evaluation 

f. Efficiency 

g. Ethics 

h. Economy 

i. Trade-offs/cost, benefit 

j. Consequences 

k. Systems thinking 

Information and 

a. Research and use of infonnation 

Communication Technology 

b. Recognizing bias 

(ICT) 

c. Extracting critical information 

d. Media literacy 

e. Use of digital infonnation systems for innovation and 
creative expression 

f. Infonnation/communication operations and concepts 

g. Use of ICT and learning technologies to leam new 
content and create knowledge (such as wikis, blogs, 
probeware, and other collaboration technologies) 

h. Data-driven decision-making 

i. Global awareness 

j . Ethics 
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Appendix C: Alignment Table - Comparing the NAEP Technology and 
Engineering Literacy Assessment Areas to U.S. Source Documents 


2014 NAEP - 
Technology and 
Society 

ISTE - National Education 
Technology Standards 
(2007) 

ITEEA - Standards for 
Technological Literacy 
(2007) 

Interaction of 
Technology and 
Humans 

Digital Citizenship: Students 
understand human, cultural, and societal 
issues related to technology and practice 
legal and ethical behavior 
5.d. Students exhibit leadership for 
digital citizenship. 

Standard 4: Students will develop an 
understanding of the cultural, social, 
economic, and political effects of 
technology. 

Standard 6: Students will develop an 
understanding of the role of society in 
the development and use of technology. 
Standard 7: Students will develop an 
understanding of the influence of 
technology on history. 

Standard 13: Students will develop 
abilities to assess the impact of products 
and systems. 

Effects of Technology 
on the Natural World 


Standard 5: Students will develop an 
understanding of the effects of 
technology on the environment. 
Standard 13: Students will develop 
abilities to assess the impact of products 
and systems. 

Effects of Technology 
on the World of 
Information and 
Knowledge 

5. Digital Citizenship: 

5.a. Students advocate and practice safe, 
legal, and responsible use of information 
and technology. 

5.b. Students exhibit a positive attitude 
toward using technology that supports 
collaboration, learning, and productivity. 

Standard 13: Students will develop 
abilities to assess the impact of products 
and systems. 

Ethics, Equity, and 
Responsibility 

5. Digital Citizenship: 

5.a. Students advocate and practice safe, 
legal, and responsible use of information 
and technology. 
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2014 NAEP - 
Design and 
Systems 

ISTE - National Education 
Technology Standards 
(2007) 

ITEEA - Standards for 
Technological Literacy 
(2007) 

Nature of Technology 

1. Creativity and Innovation: 
l.a. Students apply existing knowledge to 
generate new ideas, products, or 
processes. 

6. Technology Operations and 
Concepts: 

6.a. Students understand and use 
technology systems. 

Standard 1: Students will develop an 
understanding of the characteristics and 
scope of technology. 

Standard 2: Students will develop an 
understanding of the core concepts of 
technology. 

Standard 3: Students will develop an 
understanding of the relationships among 
technologies and the connections 
between technology and other fields of 
study. 

Engineering Design 

1. Creativity and Innovation: 
l.a. Students apply existing knowledge to 
generate new ideas, products, or 
processes. 

l.c. Students use models and simulations 
to explore complex systems and issues. 

3. Research and Information Fluency: 
3.a. Students plan strategies to guide 
inquiry. 

3. d. Students process data and report 
results. 

4. Critical Thinking, Problem-Solving, 
and Decision-Making: 

Students use critical-thinking skills to 
plan and conduct research, manage 
projects, solve problems, and make 
informed decisions using appropriate 
digital tools and resources. 

Standard 8: Students will develop an 
understanding of the attributes of design. 
Standard 9: Students will develop an 
understanding of engineering design. 
Standard 11: Students will develop 
abilities to apply the design process. 

Systems Thinking 

1. Creativity and Innovation: 
l.a. Students apply existing knowledge to 
generate new ideas, products, or 
processes. 

6. Technology Operations and 
Concepts: 

6.a. Students understand and use 
technology systems. 

Standard 13: Students will develop 
abilities to assess the impact of products 
and systems. 

Maintenance and 
Troubleshooting 

6. Technology Operations and 
Concepts: 

6.c. Students troubleshoot systems and 
applications. 

Standard 10: Students will develop an 
understanding of the role of 
troubleshooting, research and 
development, invention and innovation, 
and experimentation in problem-solving. 
Standard 12: Students will develop 
abilities to use and maintain 
technological products and systems. 
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2014 NAEP - 
Information and 
Communication 
Technology (ICT) 

ISTE - National Education 
Technology Standards 
(2007) 

ITEEA - Standards for 
Technological Literacy 
(2007) 

Construction and 
Exchange of Ideas 
and Solutions 

2. Communication and Collaboration: 

Students use digital media and 
environments to communicate and work 
collaboratively, including at a distance, to 
support individual learning and 
contribute to the learning of others. 


Information Research 

3. Research and Information Fluency: 

Students apply digital tools to gather, 
evaluate, and use information. 


Investigation of 
Problems 

4. Critical Thinking, Problem-Solving, 
and Decision-Making: Students use 
critical-thinking skills to plan and 
conduct research, manage projects, solve 
problems, and make informed decisions 
using appropriate digital tools and 
resources. 

Standard 8: Students will develop an 
understanding of the attributes of design. 
Standard 9: Students will develop an 
understanding of engineering design. 
Standard 11: Students will develop 
abilities to apply the design process. 

Acknowledgment of 
Ideas and Information 

5. Digital Citizenship: Students 
understand human, cultural, and societal 
issues related to technology and practice 
legal and ethical behavior. 


Selection and Use of 
Digital Tools 

6. Technology Operations and 
Concepts: Students demonstrate a sound 
understanding of technology concepts, 
systems, and operations. 

Standard 17: Students will develop an 
understanding of and be able to select 
and use information and communication 
technologies. 
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Appendix D: Alignment Table - Comparing the NAEP Technology and 
Engineering Literacy Assessment Areas to International Source Documents 


2014 NAEP 

International 
Baccalaureate 
Organization 
Middle Years 
Programme 
Technology 
(2006) 

EU Key 

Competences for 
Lifelong 
Learning - A 
European 
Framework 
(2007) 

UK Framework 
for Secondary 
ICT 
(2008) 

PISA Framework 
for 

Assessment of 
ICT Literacy 
(2006) 

Technology and 
Society 

Interaction of 
Technology and 
Humans 

Students identify the 
problem to be 
solved. 

Understanding the 
potential of 1ST to 
support creativity 
and innovation for 
personal fulfillment, 
social inclusion and 
employability. 

Interest in using 1ST 
to broaden horizons 
by taking part in 
communities and 
networks for 
cultural, social, and 
professional 
purposes. 



Effects of 
Technology on the 
Natural World 

Students identify the 
problem to be 
solved. 




Effects of 
Technology on the 
World of 
Information and 
Knowledge 





Ethics, Equity, and 
Responsibility 

During the course, 
students should: 

• Carry out units of 
work in technology 
using materials and 
techniques safely 
and responsibly 

• Provide evidence 
of personal 
engagement with the 
subject (motivation, 
independence, 
general 

positive attitude) 
when working in 
technology. 

Positive attitude and 
sensitivity to safe 
and responsible use 
of the Internet, 
including 
privacy issues and 
cultural differences. 
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2014 NAEP 

International 
Baccalaureate 
Organization 
Middle Years 
Programme 
Technology 
(2006) 

EU Key 

Competences for 
Lifelong 
Learning - A 
European 
Framework 
(2007) 

UK Framework 
for Secondary 
ICT 
(2008) 

PISA Framework 
for 

Assessment of 
ICT Literacy 
(2006) 

Design and 

Systems 

Nature of 
Technology 





Engineering 

Design 

Students formulate a 
design specification. 
Students design the 
product/solution. 
Students plan the 
product/solution. 
Students follow the 
plan. 

Students create the 
product/solution. 


2 Developing ideas 
2.2 Models and 
modeling 
4 Evaluating 
4.1 Evaluating work 


Maintenance and 
Troubleshooting 





Systems Thinking 
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2014 NAEP 

International 
Baccalaureate 
Organization 
Middle Years 
Programme 
Technology 
(2006) 

EU Key Competences for 
Lifelong Learning - A 
European Framework 
(2007) 

UK 

Framework 
for Secondary 
ICT 
(2008) 

PISA 

Framework 

for 

Assessment 
of ICT 
Literacy 
(2006) 

Information and 
Communication 
Technology (ICT) 

Construction and 
Exchange of Ideas 
and Solutions 

During the course, 
students should work 
effectively as 
members of a team, 
collaborating, 
acknowledging, and 
supporting the views 
of others. 

Propensity to use 1ST to work 
autonomously and in teams; 
critical and reflective attitude in 
the assessment of available 
information. 

Ability to use appropriate aids 
(presentations, graphs, charts, 
maps) to produce, present, or 
understand complex 
information. 

3 Communicating 
information 

3.1 Fitness for 
purpose 

3.2 Refining and 
presenting 
information 

3.3 

Communicating 

Processes 
- communicating 
with others 

Information 

Research 

Students should 
develop the design 
brief. 

Ability to search, collect, and 
process electronic information, 
data, and concepts and to use 
them in a systematic way. 
Ability to access and search a 
website and to use Internet- 
based services such as 
discussion forums and email. 

1 Finding 
information 

1.1 Using data and 
information 
sources 

1 .2 Searching and 
selecting 

1.3 Organizing 
and investigating 

Processes 

- accessing 
information 

- managing 
information 

- integrating 
information 

- evaluating 
information 

- constructing 
new knowledge 

Investigation of 
Problems 


Ability to use 1ST to support 
critical thinking, creativity, and 
innovation in different contexts 
at home, leisure, and work. 

2 Developing ideas 

2.1 Analyzing and 
automating 
processes 

2.2 Models and 
modeling 

2.3 Sequencing 
instructions 

Context 

- ICT for 
personal use 

- ICT for public 
use 

- ICT for 
educational 

- ICT for 
occupational 
purposes 

Acknowledgment 
of Ideas and 
Information 

During the course, 
students should work 
effectively as 
members of a team, 
collaborating, 
acknowledging, and 
supporting the views 
of others. 

Basic understanding of the 
reliability and validity of the 
information available 
(accessibility/acceptability) and 
awareness of the need to respect 
ethical principles in the 
interactive use of 1ST. 



Selection and Use 
of Digital Tools 


Understanding the main 
computer applications, including 
word processing, spreadsheets, 
databases, information storage, 
and management. 

Awareness of the opportunities 
given by the use of the Internet 
and communication via 
electronic media (email, 
videoconferencing, other 
network tools); and the 
differences between the real and 
virtual world. 


Technology 

Environments 

-Web 

- desktop 

- e-leaming 
environments 
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Appendix E: Alignment Table - Comparing ICT Subareas to the ISTE 
NETS*S and the Framework for 21st Century Learning 


A. Construction and Exchange of Ideas and Solutions 

Fourth-grade students should be able to collaborate and communicate by working with other members of a (virtual) 
team to make decisions and develop presentations using a variety of formats. Eighth-grade students should be able 
to take into account the perspectives of different audiences, use a variety of media to create effective messages, and 
modify presentations based on feedback (virtual). Twelfth-grade students should have developed strategies to be 
effective collaborators, should be able to take into account multiple viewpoints, and should be able to synthesize 
information from a variety of sources. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.1: People collaborating 
as a team can often produce 
a better product than people 
working alone. There are 
common digital tools that 
can be used to facilitate 
virtual or face-to-face 
collaboration. 

Students know that: 

1.8.1: Collaboration can take many 
forms. Pairs or teams of people can 
work together in the same space or at 
a distance, at the same time or at 
different times, and on creative 
projects or on technical tasks. 
Different communications 
technologies are used to support these 
different forms of collaboration. 

Students know that: 

1.12.1: Effective collaboration requires 
careful selection of team members, 
monitoring of progress, strategies for 
reaching agreement when there are 
opposing points of view, and iterative 
improvement of collaborative processes. 
Information and communication 
technologies can be used to record and 
share different viewpoints and to collect 
and tabulate the views of groups of people. 

Students are able to: 

1.4.2: Utilize input from 
(virtual, that is, computer- 
generated) collaborators 
and experts or sources in 
the decision-making 
process to design a product 
or presentation. 

Students are able to: 

1.8.2: Provide feedback to a (virtual) 
collaborator on a product or 
presentation, taking into account the 
other person’s goals and using 
constructive, rather than negative, 
criticism. 

Students are able to: 

1.12.2: Work through a simulation of a 
collaborative process. Negotiate team roles 
and resources, draw on the expertise and 
strengths of other team members and 
remote experts, monitor progress toward 
goals, and reflect on and refine team 
processes for achieving goals. 

1.4.3: Communicate 
information and ideas 
effectively to an audience 
in order to accomplish a 
specified purpose. 

1.8.3: Communicate information and 
ideas effectively using a variety of 
media, genres, and formats for 
multiple purposes and a variety of 
audiences. 

1.12.3: Synthesize input from multiple 
sources to communicate ideas to a variety 
of audiences using various media, genres, 
and formats. 

NETS 

•S Category 2: Communication and Collaboration 

Students use digital media and environments to communicate and work collaboratively, including at a distance, 
to support individual learning and contribute to the learning of others. Students: 

a. Interact, collaborate, and publish with peers, experts, or others employing a variety of digital 
environments and media. 

b. Communicate information and ideas effectively to multiple audiences using a variety of media and 
fonnats. 

c. Develop cultural understanding and global awareness by engaging with learners of other cultures. 

d. Contribute to project teams to produce original works or solve problems. 


A- 19 





















Technology and Engineering Literacy Framework for the 2014 NAEP 


Framework for 21st Century Learning: Communication and Collaboration 


Communicate Clearly 

• Articulate thoughts and ideas effectively using oral, written, and nonverbal communication skills in a 
variety of fonns and contexts. 

• Listen effectively to decipher meaning, including knowledge, values, attitudes, and intentions. 

• Use communication for a range of purposes (for example, to inform, instruct, motivate, and persuade). 

• Utilize multiple media and technologies, and know how to judge their effectiveness a priori as well as 
assess their impact. 

• Communicate effectively in diverse environments (including multilingual). 

Collaborate With Others 

• Demonstrate ability to work effectively and respectfully with diverse teams. 

• Exercise flexibility and willingness to be helpful in making necessary compromises to accomplish a 
common goal. 

• Assume shared responsibility for collaborative work, and value the individual contributions made by 
each team member. 
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B. Information Research 

Fourth-grade students can use digital and network tools to find information and identify sources that may be biased 
in some way. Eighth-grade students are able to use digital resources to find information and also to recognize when 
information may be distorted, exaggerated, or otherwise misrepresented. Twelfth-grade students can use advanced 
search methods and select the best digital tools and resources for various purposes, can evaluate information for 
timeliness and accuracy, and can check the credibility of sources. 

Grade 4 

Grade 8 

Grade 12 

Students know that: 

1.4.4: Digital and network tools and 
media resources are helpful for 
answering questions, but they can 
sometimes be biased or wrong. 

Students know that: 

1.8.4: Increases in the quantity of 
information available through 
electronic means and the ease by 
which knowledge can be published 
have heightened the need to check 
sources for possible distortion, 
exaggeration, or misrepresentation. 

Students know that: 

1.12.4: Advanced search techniques 
can be used with digital and network 
tools and media resources to locate 
information and to check the 
credibility and expertise of sources. 

Students are able to: 

1.4.5: Use digital and network tools 
and media resources to collect, 
organize, and display data in order 
to answer questions and solve 
problems. 

Students are able to: 

1.8.5: Select and use appropriate 
digital and network tools and media 
resources to collect, organize, 
analyze, and display supporting data 
to answer questions and test 
hypotheses. 

Students are able to: 

1.12.5: Select digital and network 
tools and media resources to gather 
information and data on a practical 
task, and justify choices based on the 
tools’ efficiency and effectiveness 
for a given purpose. 

1.4.6: Search media and digital 
sources on a community issue and 
identify sources that may be biased. 

1.8.6: Search media and digital 
resources on a community or world 
issue and identify specific examples 
of distortion, exaggeration, or 
misrepresentation of information. 

1.12.6: Search media and digital 
resources on a community or world 
issue and evaluate the timeliness and 
accuracy of the information as well 
as the credibility of the source. 

NETS*S Category 3: Research and Information Fluency 

Students apply digital tools to gather, evaluate, and use information. Students: 

a. Plan strategies to guide inquiry. 

b. Locate, organize, analyze, evaluate, synthesize, and ethically use information from a variety of 
sources and media. 

c. Evaluate and select information sources and digital tools based on the appropriateness to specific 
tasks. 

d. Process data and report results. 
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Framework for 21st Century Learning: Communication and Collaboration 


Information Literacy 

Access and Evaluate Information 

• Access information efficiently (time) and effectively (sources). 

• Evaluate information critically and competently. 

Use and Manage Information 

• Use information accurately and creatively for the issue or problem at hand. 

• Manage the flow of information from a wide variety of sources. 

• Apply a fundamental understanding of the ethical and legal issues surrounding the access and use of 
information. 
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C. Investigation of Academic and Practical Problems 

Fourth-grade students are able to use digital tools to investigate local issues, test hypotheses, and build models. 
Eighth-grade students are able to use digital tools to investigate alternative solutions to global issues, test moderately 
complicated hypotheses, build models, and conduct simulations. Twelfth-grade students can conduct more 
sophisticated investigations and simulations as well as recognize their limitations. For all levels the focus is on types 
of hardware and software rather than on use of particular hardware or software products. 

Grade 4 

Grade 8 

Grade 12 

Students are able to: 

1.4.7: Use digital tools and resources 
to identify and investigate a local 
issue and generate possible 
solutions. 

Students are able to: 

1.8.7: Use digital tools to identify a 
global issue and investigate possible 
solutions. Select and present the 
most promising sustainable solution. 

Students are able to: 

1.12.7: Use digital tools and 
resources to identify a complicated 
global issue and develop a 
systematic plan of investigation. 
Present findings in terms of pros and 
cons of two or more innovative 
sustainable solutions. 

1.4.8: Use digital tools to test simple 
hypotheses in various subject areas. 

1.8.8: Use digital tools to gather and 
display data in order to test 
hypotheses of moderate complexity 
in various subject areas. Draw and 
report conclusions consistent with 
observations. 

1.12.8: Use digital tools to collect, 
analyze, and display data in order to 
design and conduct complicated 
investigations in various subject 
areas. Explain rationale for the 
design and justify conclusions based 
on observed patterns in the data. 

1.4.9: Use digital models to describe 
how parts of a whole interact with 
each other in a model of a system. 

1.8.9: Use a digital model of a 
system to conduct a simulation. 
Explain how changes in the model 
result in different outcomes. 

1.12.9: Having conducted a 
simulation of a system using a 
digital model, draw conclusions 
about the system, or propose 
possible solutions to a problem or 
ways to reach a goal based on 
outcomes of the simulation. Critique 
the conclusions based on the 
adequacy of the model. 

NETS # S Category 4: Critical Thinking and Decision-Making 

Students use critical-thinking skills to plan and conduct research, manage projects, solve problems, and make 
informed decisions using appropriate digital tools and resources. Students: 

a. Identify and define authentic problems and significant questions for investigation. 

b. Plan and manage activities to develop a solution or complete a project. 

c. Collect and analyze data to identify solutions and/or make informed decisions. 

d. Use multiple processes and diverse perspectives to explore alternative solutions. 
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Framework for 21st Century Learning: Communication and Collaboration 


Critical Thinking and Problem-Solving 

Reason Effectively 

• Use various types of reasoning (inductive, deductive, etc.) as appropriate to the situation. 

Use Systems Thinking 

• Analyze how parts of a whole interact with each other to produce overall outcomes in complex 
systems. 

Make Judgments and Decisions 

• Effectively analyze and evaluate evidence, arguments, claims, and beliefs. 

• Analyze and evaluate major alternative points of view. 

• Synthesize and make connections between information and arguments. 

• Interpret information and draw conclusions based on the best analysis. 

• Reflect critically on learning experiences and processes. 

Solve Problems 

• Solve different kinds of nonfamiliar problems in both conventional and innovative ways. 

• Identify and ask significant questions that clarify various points of view and lead to better solutions. 
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D. Acknowledgment of Ideas and Information 

Fourth-grade students exhibit digital citizenship by understanding that it is permissible to use others’ ideas as long as 
appropriate credit is given but that copyrighted materials cannot be shared freely. Eighth-grade students should be 
aware of and comply with laws and ethical guidelines for incorporating ideas, text, and images into their own work. 
Twelfth-grade students should understand the reasons for protecting intellectual property and demonstrate 
responsible and ethical behaviors when using ideas, quotes, and images from others. 

Grade 8 

Students know that: 

1 . 8 . 10 : Style guides provide detailed 
examples for how to give 
appropriate credit to others when 
incorporating their ideas, text, or 
images in one’s own work. 


Students are able to: 

1 . 8 . 11 : Identify or provide examples 
of fair use practices that apply 
appropriate citation of sources when 
using information from books or 
digital resources. 


NETS*S Category 5: Digital Citizenship 

Students understand human, cultural, and societal issues related to technology and practice legal and ethical 
behavior. Students: 

a. Advocate and practice safe, legal, and responsible use of information and technology. 

b. Exhibit a positive attitude toward using technology that supports collaboration, learning, and 
productivity. 

c. Demonstrate personal responsibility for lifelong learning. 

d. Exhibit leadership for digital citizenship. 


Grade 12 

Students know that: 

1 . 12 . 10 : Legal requirements 
governing the use of copyrighted 
information and ethical guidelines 
for appropriate citations are intended 
to protect intellectual property. 


Students are able to: 

1 . 12 . 11 : Identify or provide 
examples of responsible and ethical 
behavior that follow the letter and 
spirit of current laws concerning 
personal and commercial uses of 
copyrighted material as well as 
accepted ethical practices when 
using verbatim quotes, images, or 
ideas generated by others. 


Grade 4 

Students know that: 

1 . 4 . 10 : It is allowable to use other 
people’s ideas in one’s own work 
provided that proper credit is given 
to the original source, whether 
information is shared in person or 
through ICT media. 

Students are able to: 

1 . 4 . 11 : Identify or provide examples 
demonstrating respect for 
copyrighted material, such as 
resisting the request from a friend to 
copy a song from a CD or placing 
copyrighted material online. 
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Framework for 21st Century Learning: Communication and Collaboration 

Life and Career Skills: Leadership and Responsibility 

Guide and Lead Others 

• Use interpersonal and problem-solving skills to influence and guide others toward a goal. 

• Leverage strengths of others to accomplish a common goal. 

• Inspire others to reach their very best via example and selflessness. 

• Demonstrate integrity and ethical behavior in using influence and power. 

Be Responsible to Others 

• Act responsibly with the interests of the larger community in mind. 
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E. Selection and Use of Digital Tools 

Fourth-grade students know that different digital tools have different purposes and are able to use a number of 
different tools. Eighth-grade students can categorize digital tools by function and can select appropriate tools and 
demonstrate effective use of the tools for different purposes. Twelfth-grade students are competent in the use of a 
broad variety of digital tools and can justify why certain tools are chosen over others that might accomplish the same 
task, by referencing specific features. 

Grade 4 Grade 8 Grade 12 

Students know that: Students know that: Students know that: 

1.4.12: Different digital tools have 1.8.12: Certain digital tools are 1.12.12: A variety of digital tools 

different purposes. appropriate for gathering, exist for a given purpose. The tools 

organizing, analyzing, and differ in features, capacities, 

presenting information, while other operating modes, and style, 
kinds of tools are appropriate for Knowledge about many different 
creating text, visualizations, and ICT tools is helpful in selecting the 

models and for communicating with best tool for a given task, 
others. 

Students are able to: Students are able to: Students are able to: 

1.4.13: Use digital tools (appropriate 1.8.13: Use appropriate digital tools 1.12.13: Demonstrate the capability 

for fourth-grade students) to accomplish a variety of tasks, to use a variety of digital tools to 

effectively for different purposes, including gathering, analyzing, and accomplish a task or develop a 

such as searching, organizing, and presenting information as well as solution for a practical problem, 

presenting information. creating text, visualizations, and Justify the choice of tools, explain 

models and communicating with why other tools were not used based 
others. on specific features of the tools, and 

summarize the results. 

NETS # S Category 6: Technology Operations and Concepts 

Students demonstrate a sound understanding of technology concepts, systems, and operations. Students: 

a. Understand and use technology systems. 

b. Select and use applications effectively and productively. 

c. Troubleshoot systems and applications. 

d. Transfer current knowledge to learning of new technologies. 


A-27 


















Technology and Engineering Literacy Framework for the 2014 NAEP 


Framework for 21st Century Learning: Communication and Collaboration 


Apply Technology’ Effectively 

• Use technology as a tool to research, organize, evaluate, and communicate information. 

• Use digital technologies (computers, PDAs, media players, GPS, etc.), communication/networking 
tools, and social networks appropriately to access, manage, integrate, evaluate, and create information 
to successfully function in a knowledge economy. 

• Apply a fundamental understanding of the ethical/legal issues surrounding the access and use of 
information technologies. 
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Appendix F: Alignment Table - Comparing Design and Systems Subareas to 
the ITEEA Standards for Technological Literacy 


2014 NAEP 

Standards for Technological 
Literacy 

A. Nature of Technology 

Fourth graders should know that 
technology involves tools, materials, and 
creative thinking used to meet human 
needs and wants. Eighth graders should 
know that technology advances through 
invention and innovation and requires a 
variety of resources. Twelfth graders 
should know how technology coevolves 
with science and other fields to allow 
people to accomplish challenging tasks. 

Standard 1: Students will develop an 
understanding of the characteristics and 
scope of technology. 

Standard 2: Students will develop an 
understanding of the core concepts of 
technology. 

Standard 3: Students will develop an 
understanding of the relationships among 
technologies and the connections between 
technology and other fields of study. 

B. Engineering Design 

Fourth graders should start to answer the 
question “How are technologies created?” 
by learning to deal with simple yet 
systematic design challenges. Eighth 
graders should be able to use a more 
elaborate engineering design process, 
including problem definition, the use of 
prototypes, testing and iteration, and trade- 
offs. Twelfth graders should have a deep 
understanding and a broad array of design 
skills, including optimization. 

Standard 8: Students will develop an 
understanding of the attributes of design. 

Standard 9: Students will develop an 
understanding of engineering design. 

Standard 11: Students will develop 
abilities to apply the design process. 
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C. Systems Thinking 

Fourth graders should be able to identify 
systems, subsystems, components, and 
boundaries in their everyday world and to 
construct simple systems designed to 
accomplish particular goals. Eighth graders 
should be able to describe goals, inputs, 
outputs, and processes of systems, to use 
reverse engineering and life cycles to 
analyze systems in terms of feedback and 
the flow of energy, and to modify and 
construct moderately complicated systems. 
Twelfth graders should understand that 
systems are embedded in larger systems, to 
recognize factors that stabilize systems, to 
use systems for forecasting, and to redesign 
complicated systems to improve reliability. 

Standard 1: Students will develop an 
understanding of the characteristics and 
scope of technology. 

Standard 2: Students will develop an 
understanding of the core concepts of 
technology. 

Standard 13: Students will develop 
abilities to assess the impact of products 
and systems. 

D. Maintenance and Troubleshooting 

Fourth graders should recognize that tools 
and machines need to be cared for and that 
devices that fail can be fixed or replaced. 
Eighth graders should know that tools and 
machines must be maintained and be able 
to use a troubleshooting process to 
diagnose problems in technological 
systems. Twelfth graders should 
understand the importance of maintenance, 
be able to analyze malfunctions, and be 
able to devise ways to reduce future 
failures. 

Standard 10: Students will develop an 
understanding of the role of 
troubleshooting, research and development, 
invention and innovation, and 
experimentation in problem-solving. 

Standard 12: Students will develop 
abilities to use and maintain technological 
products and systems. 
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Appendix G: NAEP Technology and Engineering Literacy Preliminary 
Achievement Level Definitions 

Congress authorized the National Assessment Governing Board to develop appropriate student 
achievement levels on NAEP. The achievement level definitions are statements of what students 
should know and be able to do on NAEP at grades 4, 8, and 12. To fulfill its statutory 
responsibility, the Governing Board developed a policy to guide the development of achievement 
levels for all NAEP subjects. Three levels of achievement were identified to provide the public, 
educators, and policymakers with infonnation on student perfonnance on NAEP. These levels — 
Basic, Proficient, and Advanced — are used as a primary means of reporting NAEP results to 
describe “how good is good enough” at grades 4, 8, and 12. 

Table G-l displays the Governing Board’s generic policy definitions for Basic, Proficient, and 
Advanced achievement that pertain to all NAEP subjects and grades. 

Table G-l. Generic Achievement Level Policy Definitions 
for the National Assessment of Educational Progress 


Achievement 

Level 

Policy Definition 

Advanced 

This level signifies superior performance. 

Proficient 

This level represents solid academic performance for each grade assessed. 
Students reaching this level have demonstrated competency over 
challenging subject matter, including subject matter knowledge, 
application of such knowledge to real-world situations, and analytical 
skills appropriate to the subject matter. 

Basic 

This level denotes partial mastery of prerequisite knowledge and skills 
that are fundamental for proficient work at each grade. 


During the framework development process the project committees are asked to develop 
preliminary achievement level definitions, based on the generic policy definitions, to guide item 
development. Essentially the purpose of these statements is to provide examples of what students 
performing at the Basic, Proficient, and Advanced achievement levels should know and be able 
to do in terms of the technology and engineering literacy content areas and practices identified in 
the framework. The intended audiences for these preliminary definitions are the NAEP 
assessment development contractor and item writers. The definitions are to be used to ensure that 
a broad range of items is developed at each grade level. Tables G-2, G-3, and G-4 present the 
preliminary achievement level definitions for grades 4, 8, and 12 as bullet points to clearly 
illustrate the technology and engineering literacy content and practices expected at each grade 
level. 
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The preliminary definitions include illustrative statements selected from the framework’s 
technology and engineering literacy content and practices. The statements are not intended to 
represent the entire set of objectives from the assessment targets, practices, or contexts, nor do 
the preliminary achievement level definitions denote a sense of priority or importance based on 
the statements selected. 

After the assessment is administered, broadly representative panels engage in a standard- setting 
process to detennine the achievement level cut scores on the NAEP scale. The cut scores 
represent the minimum score required for performance at each NAEP achievement level. Part of 
this standard-setting process is the development of a set of paragraphs, derived from the 
preliminary achievement level definitions, to be used in reporting the NAEP Technology and 
Engineering Literacy Assessment results to the general public and other audiences. At each grade 
level there will be paragraphs describing what students should know and be able to do at the 
Basic, Proficient, and Advanced levels in terms of the technology and engineering literacy 
content and practices identified in the framework. 

Further infonnation on NAEP achievement levels can be found at www.nagb.org. 
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Table G-2. Grade 4 Preliminary Achievement Level Definitions 



BASIC 

PROFICIENT 

ADVANCED 


Students know that: 

Students know that: 

Students know that: 

Technology and 
Society 

A new tool, product, or machine 
may have benefits and costs. 

The introduction of a new tool, 
product, or machine may change 
how people live and work. 

The introduction of a new tool, 
product, or machine usually 
brings both benefits and costs, and 
it may change how people live 
and work. 

Design and 

Systems 

Tools help people do their work, 
and certain common tools have 
particular uses. 

Different tools are better for 
different purposes. 

Different tools are better for 
different purposes because they 
have different features and 
function in different ways. 

Information 

and 

Communication 

Technology 

Digital and network tools can he 
used to answer questions. 

Digital and network tools and 
media resources are helpful for 
answering questions, but they can 
sometimes be biased or wrong. 

Digital and network tools and 
media resources are helpful for 
answering questions, and 
information-gathering can help to 
determine whether they are biased 
or wrong. 


Students are able to: 

Students are able to: 

Students are able to: 

Technology and 
Society 

Name two different technologies 
in their own lives. 

Compare the impact of two 
different technologies on their 
own lives. 

Compare the ways that their lives 
would change if one or the other 
technology were to disappear. 

Design and 
Systems 

Build a model from a kit with 
instructions. 

Build and test a model to see if it 
works as intended. 

Build and test a model to see if it 
works as intended, and use test 
data to determine how to improve 
the model. 

Information 

and 

Communication 

Technology 

Use digital and network tools and 
media resources to collect, 
organize, and display data. 

Use digital and network tools and 
media resources to collect, 
organize, and display data in order 
to answer questions and solve 
problems. 

Select and use digital and network 
tools and media resources to 
collect, organize, and display data 
in order to answer questions and 
solve problems. 
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Table G-3. Grade 8 Preliminary Achievement Level Definitions 



BASIC 

PROFICIENT 

ADVANCED 


Students know that: 

Students know that: 

Students know that: 

Technology and 
Society 

Technology interacts with society, 
sometimes bringing about 
changes in a society’s economy. 

Technology interacts with society, 
sometimes bringing about 
changes in a society’s economy 
and culture, and may lead to new 
needs and wants. 

Technology interacts with society, 
sometimes bringing about 
changes in a society’s economy, 
politics, and culture, and often 
leading to the creation of new 
needs and wants. 

Design and 

Systems 

One tool is better than another for 
a given task as a result of prior 
improvements. 

Tools have been improved over 
time to further the reach of hands, 
voices, memory, and the five 
human senses. 

Tools have been improved over 
time to further the reach of hands, 
voices, memory, and the five 
human senses, and to do so more 
efficiently, accurately, or safely. 

Information 

and 

Communication 

Technology 

Some information available 
through electronic means is 
exaggerated or wrong. 

Increases in the ease by which 
knowledge can be published have 
heightened the need to check 
sources for possible distortion, 
exaggeration, or 
misrepresentation. 

There are various effective ways 
to check information available 
through electronic means to 
identify possible distortion, 
exaggeration, or 
misrepresentation. 


Students are able to: 

Students are able to: 

Students are able to: 

Technology and 
Society 

Identify the impacts of a given 
technology in a society. 

Identify the impacts of a given 
technology in a society, and 
predict how it might affect a 
different society. 

Compare the impacts of a given 
technology on two different 
societies, noting factors that may 
make a technology appropriate 
and sustainable in one society but 
not in another. 

Design and 

Systems 

Design and build a simple model 
that meets a requirement. 

Design and build a simple model 
that meets a requirement, fix it 
until it works (iteration), test it, 
and gather and display data that 
describe its properties using 
graphs and tables. 

Design and build a simple model 
that meets a requirement, fix it 
until it works (iteration), test it, 
and gather and display data that 
describe its properties using 
graphs and tables. Interpret the 
results to suggest improvements 
and new designs. 

Information 

and 

Communication 

Technology 

Select and use appropriate digital 
and network tools and media 
resources to collect, organize, and 
display data. 

Select and use appropriate digital 
and network tools and media 
resources to collect, organize, 
analyze, and display supporting 
data to answer simple questions 
and test basic hypotheses. 

Select and use appropriate digital 
and network tools and media 
resources to collect, organize, 
analyze, and display supporting 
data to answer complicated 
questions and test hypotheses, and 
explain how to go about this to 
others working on the same 
project. 
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Table G-4. Grade 12 Preliminary Achievement Level Definitions 



BASIC 

PROFICIENT 

ADVANCED 


Students know that: 

Students know that: 

Students know that: 

Technology and 
Society 

Changes caused by the 
introduction and use of a new 
technology can be gradual or 
rapid as well as big or small. 

Changes caused by the 
introduction and use of a new 
technology can range from 
gradual to rapid and from subtle 
to obvious, and can change over 
time. 

Changes caused by the 
introduction and use of a new 
technology can range from 
gradual to rapid and from subtle 
to obvious and can change over 
time. These changes may vary 
from society to society as a result 
of differences in a society’s 
economy, politics, or culture. 

Design and 

Systems 

The development of tools has 
influenced and advanced society. 

The evolution of tools and 
materials has played an essential 
role in the development and 
advancement of cities and 
industrial societies. 

The evolution of tools and 
materials has played an essential 
role in the advancement of 
civilization, from the 
establishment of cities and 
industrial societies to today’s 
global trade and commerce 
networks. 

Information and 
Communication 
Technology 

Advanced search techniques can 
be used with digital and network 
tools and media resources to 
locate information. 

Advanced search techniques can 
he used with digital and network 
tools and media resources to 
locate information and to check 
the credibility and expertise of 
sources. 

Advanced search techniques can 
he used with digital and network 
tools and media resources to 
locate different kinds of 
information and to check 
credibility and expertise of 
different types of sources. 


Students are able to: 

Students are able to: 

Students are able to: 

Technology and 
Society 

Analyze the cultural, social, 
economic, and/or political 
changes that may he triggered by 
the introduction of a specific 
technology into a society. 

Analyze cultural, social, 
economic, and/or political 
changes that may he triggered by 
the transfer of a specific 
technology from one society to 
another. 

Analyze cultural, social, 
economic, and/or political 
changes that may be triggered by 
the transfer of a specific 
technology from one society to 
another. Include anticipated and 
unanticipated effects. 

Design and 

Systems 

Construct and test a model to see 
if it meets the requirements of a 
problem, then suggest 
improvements. 

Construct and test several models 
to determine which is best in 
meeting the requirements of a 
problem. 

Construct and test several models 
to see if they meet the 
requirements of a problem. 
Combine features to achieve the 
best solution. 

Information and 
Communication 
Technology 

Select digital and network tools 
and media resources to gather 
information and data on a 
practical task. 

Select digital and network tools 
and media resources to gather 
information and data on a 
practical task, and justify choices 
based on the tools’ efficiency and 
effectiveness for a given purpose. 

Select digital and network tools 
and media resources to gather 
information and data on a 
practical task, justify choices 
based on the tools’ efficiency and 
effectiveness for a given purpose, 
and advise others on which tools 
and resources would best meet 
their needs. 
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